When a repair agent runs a check and sees it pass, everybody treats that as evidence the reported bug is fixed. This paper makes that testable: capture every validation the agent runs, strip it to a test-only patch, and replay it against the original broken code, the agent's candidate, and the developer's real fix. Across 3,730 validation events in 643 runs, 46.0% of passing checks carry no information about the bug at all — they pass on the broken code too. And 23.8% of runs submit a patch whose entire positive evidence base is that kind of check.
An agent fixing a bug runs commands and reads outcomes. A passing command feels like progress, and the agent's own summary will describe it as confirmation. But a check only says something about the bug if it would have failed before the fix. If it passes on the broken code as well, the agent has learned that nothing is on fire — not that anything is repaired. Step through three checks an agent actually ran and compare what it concluded with what a replay shows.
Every one of these is a green result the agent counted toward closing the task. The difference between them is invisible in the transcript, invisible in the summary, and invisible in CI. It becomes visible only when you re-run the same check against code you already know is broken.
The method is mechanical and, once you see it, obvious. Intercept every command the agent runs after its first real edit. Split the working tree into the production change and the test-only change. Then take the test-only part and replay it, deterministically, against three states: B, the original buggy code; S, whatever the agent has now; and G, the developer's actual fix. B and G together decide what the check was worth. Flip the two outcomes below and watch the role change.
Only one of the four cells is what everyone assumes they are getting. B fails, G passes means the check detects something present in the broken code and absent from both the candidate and the real fix — that is evidence. B passes, G passes means the check was green before the bug was ever touched, so it is a regression guard wearing a fix's clothes. The rarest cell, B passes and G fails, is actively wrong: a property the buggy code satisfies and the correct repair does not.
Run that classification over every validation event in the study and the shape is stark. Of the 2,548 events that both passed and could be compared, 1,171 — 46.0% — were regression-only or outright misleading. That is not a tail. That is nearly half of everything the agents were treating as confirmation. Toggle between the full event population and the passing-and-comparable subset the headline is computed over.
The encouraging half of the picture: 72.9% of baseline runs do contain at least one genuinely bug-discriminating event. The agents are not blind. The problem is that they cannot tell which of their green results was the useful one, so they stop as readily on the hollow ones.
Knowing the failure exists is not the same as fixing it, so the authors ran a pre-registered three-arm experiment. One arm is the plain agent. One is an attention-matched nudge — a message after every passing check asking it to reconsider, carrying no new information. The third replays the check on the broken code in real time and injects the actual outcome. The third arm is the one that isolates whether the information matters or just the interruption.
The honest part is the ceiling the authors drew themselves. They pre-registered 10 points as the smallest effect they would call practically meaningful, and both headline estimates land under it: −7.8 and +7.4. The effects are real by the statistics and undersized by their own standard. Roughly a third of the improvement comes from the bare reminder, and across two replications — a different scaffold, then a different model — the extra information on top of the reminder only showed up in one configuration.
The measurement rig here is expensive — deterministic replay against a known-good fix you would not have in production. But the cheap half generalises. You do not need the developer's gold patch to run the single most valuable replay: take the test that just went green, check it out against the pre-fix commit, and confirm it fails. That one command converts a hollow check into a discriminating one. Point the paper's rates at your own volume.
Two things to take away for your own agent loop. First, the check is worth more than the transcript: an agent's own account of why it is confident is not a signal you can grade, but "did this test fail before the patch" is a shell command. Second, note where the intervention plateaued — the authors could only move this so far by telling the agent, which is the same lesson as the harness papers. If you want the guarantee, put the pre-fix replay in the pipeline rather than in the prompt.