A screenshot is a lossy rendering of what a program actually holds. Two different underlying states — a live formula and a number typed over the top, a saved file and an unsaved buffer — can produce identical pixels, and an agent driven by screenshots cannot tell them apart. Give the main agent code instead, so it reads and writes the real artifacts, and hand off to a click-and-screenshot subagent only for the few subgoals nothing else reaches: 28 of 108 tasks, about 1.1% of its steps. That lifted fully-correct tasks from 20.6% to 26.9% at roughly a ninth of the cost. Take the screen away entirely, though, and it scores below the screenshot baseline — the win is the split, not the purge.
The screen is a projection. Rendering throws away the thing that decides whether the task is actually done — whether that total is a formula or a literal, whether the file reached the path you named, whether the event made it into the calendar store. An agent that only ever sees pixels is reading a picture of the answer and reporting on the picture. Step through three cases and switch which lens the agent gets.
In every case the screenshot check passes and is wrong. The state check fails and is right. Nothing about the model changed between the two columns — only what it was allowed to look at.
The obvious correction is to throw the screen away and drive everything through code. That loses. Some subgoals are irreducibly visual — dragging on a canvas, dismissing a modal, reading a value that exists only as rendered output — and an agent with no way to see them stalls. The paper's harness keeps a screenshot-and-click subagent on call and routes to it rarely. Slide the routing share and watch the curve: both ends are worse than the operating point near the left edge.
The code-only variant is the instructive one. Removing the screen entirely costs 8.9 points against the screenshot baseline it was supposed to beat — so the gain is not "code good, pixels bad". It is that the main agent should work on state, with the screen reachable when the state genuinely is not.
Direct access to state buys one more thing: an independent way to check the work. When the agent calls finish, a separate checker spawns that never sees the message history, the plan, or the agent's reasoning about why it succeeded. It gets the task instructions and a shell, and it goes and looks at the artifact — is the file at the named path, does the cell hold a formula, did the event land in the store. Narration-blind, state-grounded, and capped at three retries. Switch each component off to see what it was holding up.
The gate is precise and shy: when it rejects, it is right 8 times out of 9, and it wrongly blocks a genuine success only once. But it only catches 10.5% of the runs that were not perfect — which is the subject of the next section.
Of the 76 imperfect runs that reached the gate, it waved 68 through. That looks like a broken verifier; it is not. Structural defects — output missing, unsaved, written to the wrong path, in the wrong format — are exactly what an independent look at the artifact can settle. Value errors cannot be settled that way: the checker reads the same source the agent read, interprets it the same way, and arrives at the same wrong number. Shared interpretation, not gullibility. Toggle the error class to see which side of the line each failure falls on.
Nearly half of all imperfect runs are the agent computing the wrong value or misreading the instruction — 38 of 79. State-grounding moved the bottleneck off perception and onto reasoning, which is progress, but it means the next win is not another verifier.
The lesson transfers to anything with a real path to the artifact — a database row, an API, a document store, the DOM — not just desktops. Set your daily task volume and how much of your work is genuinely render-only (image edits, layout, chart appearance: things where the visual outcome is the deliverable), then switch the architecture. The more of your workload is irreducibly visual, the less the split buys you. Treat the totals as illustrative; the per-task anchors are the paper's.
The shape of the answer matters more than the totals: the cost gap is large and roughly fixed, while the accuracy gap narrows as your workload gets more visual. If most of what your agent touches is addressable through an API or a file, the split is close to free money. If most of it is genuinely pixels, this paper is not about you.