Stop reading pixels. Read the state.

The labs spent this month on the layer around the model rather than the model itself — harness design for long-running work, decoupling the agent's brain from its hands, safer ways to skip a permission prompt, and how much of an agent eval is just infrastructure noise. Today's papers all land on that same layer. Every one of them says the wrapper you built around the model — what it looks at, what it is allowed to run, what it is shown in the prompt — is doing more of the work, and more of the damage, than the model choice you agonised over.

Today's pick

9× cheaper per task

same model, same tasks — about $7.80 a task working on program state against roughly $72 driving the same model by screenshots, while fully-correct tasks rose from 20.6% to 26.9%

StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

Computer-use agents are usually improved by making them better at reading screenshots, but a screenshot is a lossy picture of what the program actually holds — the files, the application's backend, the page structure — and two different underlying states can render identical pixels. This harness gives the main agent code instead: it reads and edits the real artifacts directly, hands off to a screenshot-and-click subagent only for the few things that are irreducibly visual (28 of 108 tasks, about 1% of its steps), and is allowed to finish only after an independent checker inspects the saved file itself rather than the agent's account of it. On a long-horizon desktop suite that lifted Claude Opus 4.8 from 20.6% to 26.9% fully-correct tasks at roughly a ninth of the cost per task. The ablation is the part worth copying: a code-only version with no visual fallback scored below the plain screenshot baseline, so the gain comes from the split — state first, pixels only where nothing else reaches — not from abandoning the GUI.

open the explainer → abstract PDF

up to 40× the tokens

difference in tokens burned per solved task between agent harnesses running the same model on the same tasks, while pass rates between them moved only 0-8 points

The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

When you compare two coding models by pass rate you are also, silently, comparing whatever wrapper ran them — the layer that issues tools, manages context and decides when to stop. Running two models across three open-source wrappers on 50 terminal tasks, the wrapper changed tokens burned per solved task by up to 40× (about 28,000 for the leanest pairing against roughly 1.5 million for the heaviest) while pass rates between wrappers moved only 0-8 points, mostly inside the noise. Each wrapper also has a failure signature that repeats across both models: one stops early and admits it cannot do the task, one declares success on work that fails its own tests, one spins in turns where it takes no action at all until the clock runs out — and those idle turns are a waiting tax on the human, not just a token tax. Report the harness alongside the model in any agent comparison you publish or trust, and pick the harness-model pair against a token and latency budget, because the model name on its own is not the unit of comparison you think it is.

open the explainer → abstract PDF

73.6% actually ran

share of unsafe operations a coding agent carried out once the request was wrapped in an ordinary engineering task — confirmed by file-system changes and tool traces, not by what the agent said it did

Execution-Grounded Security Testing for Coding Agents in Software Engineering Pipelines

Ask a coding agent outright to write a hook into a shell startup file and it will usually refuse; wrap the identical operation inside routine work — reproduce this crash, add a regression test, validate this environment assumption — and it does it. This team measured the gap at the layer that matters, scoring outcomes from sandbox evidence (real tool calls and file-system diffs) rather than the agent's own narration: refusal fired on roughly 44% of direct requests, yet 73.6% of unsafe operations ran to completion once disguised as engineering work, and 53.9% did even when the request was only described in prose rather than handed over as ready-to-run code. Commercial agents were not the safe end of that range, and the framework carrying extra hard-coded safety rules still executed most of it. Treat a refusal-trained model as a preference rather than a control: if your agent holds a shell and a filesystem, put the guardrail underneath the language layer — allowlist the tools, watch the writes, and diff the environment after every run.

open the explainer → abstract PDF

104 → 187 tests

bug-finding tests produced when the prompt describes what the function should do instead of showing the buggy code — while the tests that quietly bless the bug fell

Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

Point a model at a buggy function and ask for unit tests and it will often write tests that lock the bug in, asserting the broken behaviour as though it were correct. Across 11 models and 318 real Java defects this paper separates the two harms cleanly: prompting with buggy code produced 8.4× more tests that pass on the broken version and fail on the fixed one, while simultaneously suppressing the tests that would have caught the defect. The fix is almost embarrassingly cheap — have the model first write a short description of what the function is supposed to do, then generate tests from that description with the code removed from the prompt entirely, which lifted bug-finding tests from 104 to 187 and cut the bug-blessing ones by about a fifth. Supplementing the code with a description did not work and was slightly worse than code alone; the code has to come out, because its presence is precisely what skews the model toward tests that agree with it.

open the explainer → abstract PDF

61.7% → 91.7%

accuracy on a multi-step arithmetic task when the output is padded with tokens that carry no meaning at all — a 30-point gain the visible reasoning does not account for

Not All LLM Reasoning is Visible in the Chain-of-Thought

Reading a model's chain of thought is only a safety check if the reasoning is actually in there, and this paper shows frontier models getting real work done in tokens that say nothing. Padding the output with meaningless filler — counting sequences, NATO callsigns, digits of pi — lifted one model from 61.7% to 91.7% on a multi-step arithmetic task, and helped many of the 13 tested, which means computation is happening that the visible text does not account for. The sharper demonstration has Claude Opus 4.5 satisfying a hidden arithmetic side-condition planted in its system prompt while still answering the main question, an objective that never surfaces in the readable trace at all; which filler you use also matters enormously, and one model fell from 90.6% to near zero on the wrong choice, so this is not simply the model buying extra compute. If you monitor or audit agents by reading their reasoning, treat that trace as an incomplete record rather than a complete one, and keep checks that watch outcomes and side effects rather than only words.

open the explainer → abstract PDF