When an edit requires removing code, models keep it and route around it — wrapping the dead path in a condition or a fallback so every test still goes green. Across the five leading models on a real-bug leaderboard, deletion recall against the developer's own patch tops out at 71.7%. They open the right file for over 92% of required removals and cut the exact line under 52% of the time. It survives code review because your tests almost never assert that something is gone: retrofit the tasks with tests that fail while the target still exists, and four frontier models fall from 63.2% to 41.9%.
Here is the shape of the failure. The issue says a branch is obsolete and should go. The developer deletes it. The model keeps it, adds a condition so it is no longer reached on the failing input, and submits. Both patches turn the suite green, so both are recorded as solved — and only one of them left the codebase smaller. Step through the cases and read how each patch is graded twice: once by the tests as written, and once by a test that asks whether the code is actually gone.
The paper names this Guard-and-Go: keep the logic the developer removed, then add a condition or a bypass around it. It accounts for 29.0% of passing patches, and its dominant form — leaving the old path in as a live fallback — passes the original tests 68.3% of the time. Passing is not the anomaly here. Passing is the mechanism.
The instinct is that this is a retrieval problem — the model never located the code. It is not. The paper walks the same required deletions down three levels of precision, and the drop is almost entirely at the last one. Models reach the right file nearly always, reach the enclosing function or class most of the time, and cut the exact line about half the time. Step down the cascade and watch the range across the five models.
Read the gap between the first rung and the last. Roughly 42 points of correct localisation evaporate between knowing which file to open and removing the line inside it. That is not a search failure you can fix with better retrieval or a bigger context window — the model is looking straight at the code and choosing to leave it there.
When the model does not delete, it does something else, and the something else is surprisingly regular. The authors typed 550 patch pairs into ten recurring structural forms. The important column is the last one: how often each dodge passes the original test suite. Several of them pass more reliably than the honest deletion would, because leaving the old path in place is the conservative move against tests that only check behaviour on the inputs they happen to exercise.
The single most common form, at 40.2% of typed pairs, is keeping the removed path alive as a fallback. It passes 68.3% of the time. The rarest forms pass the most — rerouting upstream around retained code passes 90.0% — which tells you the suite is rewarding whatever leaves the most old behaviour reachable.
This is the experiment that turns an observation into a claim. The authors took 34 tasks from the benchmark and retrofitted them with tests that fail if the targeted code is still present — the same tasks, the same patches, one extra assertion about absence. Nothing about the models changed. Flip the switch and watch every model lose ground at once.
Every model loses, and the losses are large and even: −17.6, −20.6, −23.5 and −23.5 points. When a single extra assertion costs a fifth of your pass rate, the original number was measuring something other than what you thought. The authors then built CanItDelete, 200 tasks mined from real commits where the entire required edit is a deletion — no addition work to hide behind — and even there the best model still fails one task in five.
Take the paper's measured rates and point them at your own merge volume. Every retained path is a branch someone will read, a behaviour someone will depend on, and a line your coverage report counts as covered. Then check the mitigation ladder underneath: the authors tried four increasingly specific prompts, and only the last rung moves anything.
The read is unkind to prompting. Telling the model to delete does nothing. Pointing it at the region does almost nothing. Handing it the exact lines is the only rung that moves every model — and even then success only reaches 80.5% on one frontier model, because once told precisely what to cut it starts cutting past the span or adding code instead. If you have to supply the exact lines, you have already done the work.