Hand a coding agent tools to check its own work and what comes back is not proportional to how many tools you handed it. It is proportional to how much of the way the application actually fails those tools can observe. Holding the agent, the prompt and the task fixed and changing only the tool list across 1,116 builds, one build in seven never launched at all with no tools; a single probe that installs dependencies and reports whether the server starts took that from 86.5% to 99.5%, at 214k median tokens against a full shell's 615k — the one configuration cheaper than building blind. Above it, each rung stops buying. Screenshots, the most expensive rung, help where mistakes are visible and land below a plain shell on the one task whose failure is a stutter rather than a sight.
The design is the point. A minimal coding agent was built so that the tool list is the only thing that changes between conditions — same system prompt, same models, same seven applications. Six models each built every application four or five times under each configuration, 1,116 builds in total, and a human graded every one of them blind to which condition produced it.
The rungs go: nothing at all, a linter that reads code without running it, a probe that installs dependencies and reports whether the server starts, a full unrestricted shell, and the shell plus screenshots. Three outcomes were measured — whether the application comes up at all, whether its behaviour is correct over the API, and what a human makes of the interface.
Every outcome improves as the surface grows, but each stops improving at a different rung. Survival stops first: the probe lifts launches from 86.5% to 99.5% and nothing above it moves that again, because once nearly everything starts there are no launch failures left to prevent. Machine-checked behaviour stops next — ten of the twelve points separating blind builds from shell builds already arrive with the probe alone. Only the human's judgement of the interface is still climbing at the top, and that last step is the one the statistics do not confirm.
The probe is the smallest verification in the study. It installs dependencies, starts the application's server, reports whether it listens, and repairs nothing itself. That single bit was enough: 191 of 192 builds came up, against 166 of 192 with no tools. The gain lands exactly where it is needed — the models that already launched reliably had nothing to gain, while one model gained 50 points of survival and another 20.
It is also the cheapest configuration in the study, cheaper than handing the agent nothing. The saving is not that it writes less code. The whole conversation is re-sent to the model on every step, so input dominates the bill, and a run that knows whether it has a working application takes fewer steps and re-reads less of itself. Comparing only the builds that did start, the probe still comes out about 49k tokens cheaper than working blind.
Its boundary is crisp, though: listens is not works. The one build the probe failed to save had a backend that started and accepted connections and a parse error in a frontend file, so the page could never render. Two more graded builds show the same edge at request time — an SQL type error that crashes an endpoint on first call, and a permanently false error banner, both walking straight past a check that only asks whether the server started.
This is the study's whole finding, and it explains every result above and below it. Line up the ways a web application fails against what each tool can observe, and the gains land on the diagonal. Nowhere else.
The linter is the clean demonstration. Ten builds in the study shipped a frontend that could never render, every one of them because of a parse error in a frontend file — precisely the mistake a linter exists to catch. Five of those ten came from the linter condition. Granting a tool is not the same as benefiting from it: one build called the linter once, got an error back because it had never wired the tool up, and finished without ever checking again.
The reversal is the same rule seen from the other side. On the calendar view, whose mistakes are misplaced events and interactions that do not work, screenshots score 97.3 against a plain shell's 86.4. On the log explorer, whose mistakes are lag and stutter over a 100,000-row list, the screenshot condition falls to 91.7 against the shell's 93.8, because a still image cannot show jank. That task was put in the study as a negative control, and it behaved exactly as predicted.
The paper pre-registered six primary comparisons and corrected across all six; only three survive. The shell beats no tools on behaviour by 12.26 points. The probe beats no tools on survival by 13 percentage points. The shell costs 2.35 times as much. The screenshot step does not make it: pooled over the two visually graded tasks it is worth 6.88 points, 95% interval [+0.83, +13.41], uncorrected p = .041 — and .083 once the family of six is accounted for.
Use and benefit do not line up. The largest gain in the entire sight comparison belongs to one model that took a single screenshot on the calendar task and none at all on the dashboard — so its gain cannot have come from looking, and the pooled average cannot be credited to seeing.
One more result worth having: the most expensive configuration in the study is the one that declares itself done and only then unlocks a shell. Every one of those 60 runs used the shell, 45 of them changed their own code afterwards, and the arm cost 1,199k median tokens — nearly double a shell granted up front. The 15 builds that finished unchanged scored about seven functional points higher than the 45 that needed repairs, so an agent that checks and finds nothing is itself a mild signal of a better build.
Translate the ladder to whatever your agent ships. Tick the ways your artifact actually fails in production and the panel names the cheapest set of checks whose reach covers them, priced with this study's measured cost multipliers against a no-tools baseline.
The order matters more than the total. Almost every agent should start with the equivalent of the does-it-start probe — the one bit that says the thing you just built comes up — because here it was both the largest single quality step and cheaper than not checking at all. Everything above it should be bought against a specific failure you can name.