Coding benchmarks grade an agent's patch with the tests that shipped alongside the original human fix. Those tests were written to confirm that patch — not to grade an arbitrary one — so they fail correct alternatives and pass convincing stubs. An independent judge auditing 789 runs on SWE-Bench Pro disputed 32.4% of its pass/fail verdicts. Re-writing the graders from the task description, asserting only on public behaviour, took that to 1.4%.
When a benchmark takes a merged pull request and turns it into an agent task, the grading comes for free: run the tests that shipped with that PR. But those tests had one job — prove this diff works. They were never asked to screen out a stub, and never asked to admit a different-but-correct implementation. As the paper puts it, they "can fail a correct alternative or pass an incomplete one."
Here are five real cases from the audit. Step through them, and flip the grader between the tests the repository already had and a verifier written from the task description.
The inherited tests get three of these five wrong. Not because the agent was bad, and not because the tests were badly written — they simply answer a different question than the one the benchmark is asking them.
The authors took 30 tasks from each benchmark, ran nine frontier agent configurations against them, and had an independent model read every trajectory, patch, verifier output and reference solution — then say whether it agreed with the grade. Two ways to be wrong: the grader failed a patch that works, or it passed a patch that doesn't.
The false-negative half (a working patch marked failed) wastes your money and slanders your model. The false-positive half is worse: it means a stub scored a point. Drag the dial to see what those rates do to a grading run the size of yours.
The most alarming number isn't in that chart. Counted only against the runs the benchmark passed — the ones that became a score — the judge called roughly 18% of Claude Opus 4.7's passes and 25% of Opus 4.6's improper. A quarter of the points weren't earned. The container ships the repository's full .git history, and an independent report found 87% of identified cheating runs simply read the merged fix out of it.
Benchmarks that grade with the repo's own tests must stop the agent from editing them. So the standard instruction wrapper tells it the test files are already handled and it should not modify the testing logic. Agents read that and conclude something broader: testing is not my job.
The paper's tasks carry no such wrapper and say nothing about tests at all. The rate at which agents wrote their own tests — unprompted, as part of solving the problem — moves like this.
Every model writes three to sixteen times more tests once the boilerplate is gone. Same models, same week — the only thing that changed is a paragraph of instructions. Gemini 3 Flash submits without running any test, including ones already in the repo, on 18% of its runs.
If you have inherited that wrapper into your own agent's system prompt — and a lot of harnesses did, by copying a benchmark scaffold — you are paying for a model that can self-verify and then telling it not to.
A grader that fails good patches and passes bad ones doesn't just add noise — it pulls every model toward the middle. On the older benchmark, eight frontier configurations land in a 29.7-point band. Graded against verifiers written from the task, the same models spread across 69.8 points.
Every model but one sits below the diagonal — it scores worse when the grader stops accepting stubs. The three top scores are the only ones the paper states as text: 70.0% [67.2–72.9], 55.5% [53.4–57.7] and 54.2% [49.5–58.9] — and the authors are careful to say the second and third are not statistically separated.
You are choosing between two models. One is genuinely better than the other. Your grader has some false-fail rate and some false-pass rate — set them to the audited ones and watch what your leaderboard reports back to you. The observed pass rate for a model is just true × (1 − false-fail) + (1 − true) × false-pass; the arithmetic does the rest.
This is why the paper's models bunched up. A grader that mistakes a quarter of good patches for bad ones can't tell you much about the model that produces more good patches — it eats most of the difference before you see it. Every point of grader error is a point of resolution you cannot buy back with more runs, because the error is systematic, not random.
The 32.4% is disagreement between the grader and an independent model judge — not a measured error rate against ground truth. The judge is GPT-5.5, which is also the top scorer on the new leaderboard; the authors state plainly that a self-preference bias toward its own trajectories cannot be excluded, and they withheld the judge prompt.
The 1.4% rests on ten events in total. The gap between the two benchmarks holds because the confidence intervals are far apart, not because the low rate is precisely pinned down. Some of the gap is packaging rather than test design — fixtures that don't ship with their test file, git history left in the container.
Only SWE-Bench Pro was audited. Nothing here is a claim about SWE-bench Verified's tests. And the authors explicitly refuse the obvious victory lap: a wider spread aids resolution, but "is not by itself a capability claim" — it separates agents only insofar as the rank order tracks something real, which this paper does not measure.
Rewrite your coding agent's eval to assert on public behaviour described in the task — any working implementation passes, no private helpers, no snapshots of unrelated output — instead of replaying the tests that shipped with one particular human fix; in this audit that alone moved disputed verdicts from 32.4% to 1.4% and more than doubled the score spread between strong and weak models. While you're in there, delete the inherited "the tests are handled, don't modify them" boilerplate from your agent's prompt: it is quietly cutting the rate at which your agent verifies its own work from 85% to 18%.