Every agent benchmark you have seen scores one thing: did the patch make the failing test pass. That is not the bar a repository actually applies. This benchmark mines real pull request review comments for the requirements that decide whether a patch gets merged — raise this exception type, not that one; keep the keyword arguments in order; make it safe to run twice — and gives each of 303 repair tasks a second test suite for the requirement alongside the functional one. Across four models, 644 patches passed the functional tests and 221 of them, 34.3%, broke the requirement they had been handed. The strongest model in the set hid a violation in 29.5% of its green runs. If your merge gate is a green test run, that is the share of it you are not checking.
The construction is the whole idea. Each task carries an issue description, a defect injected into a real repository, a functional test that checks the defect is gone, a requirement written in plain language, and a second test that checks the requirement is met. It also ships two reference patches: a gold one that passes both suites, and a non-compliant one that passes the functional suite and fails the requirement — the exact artefact a conventional benchmark would score as a win.
Read the middle bar of each model carefully, because it is the one that does not behave like a capability score. The rate at which a model's already-passing patches also satisfy the requirement sits between 64% and 71% for the three serious backends, and it barely tracks how good the model is at the repair itself. The strongest model resolves nearly twice as many issues as the weakest serious one and complies at almost the same rate. Getting the fix right and getting the fix acceptable are close to separate skills.
The requirements are not invented. They are extracted from real review comments in two passes: first a decomposition into atomic records — what the reviewer objected to, what they asked for, why — then a filter that throws away anything not machine-checkable. Workflow requests, documentation, formatting: gone. What survives has to be visible in behaviour, supported by the diff that answered the review, additional to functional correctness, and testable by its own separate oracle.
The two biggest categories are also the two least visible in a diff. Half the instances turn on error semantics — whether the right exception type comes out of the right branch with the right message — and just under half on schema, metadata and typing. Those are the things a reviewer notices in ten seconds and a functional test, written to check that the bug is gone, never looks at.
The obvious response is to put the requirement in the prompt, and the paper runs exactly that ablation: the same tasks with the requirement described to the agent, and without. Compliance rises everywhere, by 10 to 26 points. But the functional pass rate falls everywhere too, by up to 10 points on one model. A longer specification is a harder task, and the agent spends attention on the requirement that it was previously spending on the bug.
The trade is worth it on every model — joint success, the only number that means "mergeable", rises by 1.0 to 11.5 points — but it is a trade, not a free win, and the shape of it matters for how you deploy. The largest compliance gain, 25.6 points, belongs to the weakest model, whose functional rate collapses from 15.8% to 9.2% at the same time. Telling a model what "good" means helps most where it can least afford the attention.
Break compliance down by what the reviewer was asking for and a pattern falls out. Agents do best on requirements that live at the point of the fix: distinguishing a missing value from an empty one, preserving argument order, keeping a deprecation path intact. They do worst on requirements that ask them to look up from the defect — generalise the fix beyond the reported case, clean up a resource on every path, get the escaping right for inputs the issue never mentioned.
Scope generalisation is the clearest case, and the most instructive. On the strongest model it has the second-highest functional pass rate in the whole table, 87.1% — the agent fixes the reported bug more reliably here than almost anywhere else — and one of the lowest compliance rates, 63.0%. It solves the case it was shown and stops. That is not a reasoning failure; it is a specification failure, and the specification is missing because the person who would have written it does not read the patch until after your gate has passed it.
The useful reading is not "agents are worse than you thought". Their functional numbers are unchanged; the paper adds a second measurement, and it is a measurement your own pipeline is also not taking. Whatever share of your merge decisions rests on a green test run is a share resting on a check that was never written to answer the question you are asking it.
The fix the paper actually supports is cheap: the requirements it mines are already written down, in your own review history. Pick a request and compare the two setups.
These scenarios are illustrative — the paper measures a benchmark, not a team's pipeline. What it measures is the premise: across 644 patches that passed the functional tests, 221 broke a requirement that a human had already stated in a review comment, and stating it up front moved compliance by 10 to 26 points.