Your coding agent will not delete

Four of today's five papers are about the same uncomfortable thing: the signal you have been taking as proof. A patch goes green because no test ever checked that the old code was gone. A repair agent closes on a passing check that passes on the unfixed code too. A reflection pass reports a revision that carries no new information. An agent commits to an edit before it has looked at anything that would justify it. The fifth moves the problem upstream to your inputs — the format you declare your tools in quietly changes whether the model refuses at all.

Today's pick

63.2% → 41.9%

tasks solved, once the tests actually fail when the code that should have been removed is still there

To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

Ask a coding agent to remove something and it will usually keep the code and route around it — wrapping the old path in a condition or a fallback so everything still passes. Across the five leading models on a real-bug benchmark, deletion recall against the developer's own patch tops out at 71.7%: they reach the right file for over 92% of required removals but cut the exact line under 52% of the time, and 29.0% of passing patches use this keep-it-and-guard-it move. It survives review because the original tests almost never assert that something is gone — retrofit 34 tasks with tests that fail if the targeted code remains and four frontier models drop from 63.2% to 41.9%. If an agent writes your patches, add one test per change that fails while the old path is still reachable, because a green run is not currently evidence the removal happened.

open the explainer → abstract PDF

46% of green checks

share of passing validations that carry no information about the reported bug — they pass on the broken code as well

Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

When a repair agent runs a test and sees it pass, that result gets treated as evidence about the defect. This paper makes that testable by replaying every passing check three ways — on the original buggy code, on the agent's candidate, and on the developer's real fix — so a check either distinguishes the bug or it does not. Across 3,730 validation events in 643 runs, 46.0% of passing comparable checks carry no bug-discriminating information at all, confirming only that nothing regressed, and 23.8% of runs submit a patch whose entire positive evidence base is that kind of check. The move to copy is cheap: before trusting a passing suite, replay the same test against the pre-fix code and confirm it fails.

open the explainer → abstract PDF

about zero information gained

what a self-review pass adds on questions with a fixed answer; on judgment calls it goes negative

Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds

Nearly every agent loop now asks the model to review its own answer and revise, and this paper measures whether that second pass adds anything at all. Running the same two-pass protocol over humans and models, it finds that on objective questions model revision moves the answer no closer to the truth than simply sampling again, and on subjective ones it actively moves away — one model lost 29.2% on a film-rating task — while humans gained in both settings (+6.0% and +17.8%). Swapping the sources localises the fault to the revision step rather than the input: models degrade good human answers too. The structural reason is the part to act on — without new external information, a model conditioning on its own output cannot reduce its uncertainty, so a reflection step with no tool call, no retrieval and no second model in it is re-generation with extra latency.

open the explainer → abstract PDF

+4.8 to +11.8 pts

more tasks solved first try when edits are blocked until the agent has looked at the evidence — at up to 12.1% fewer tokens

Preventing Premature Commitment in Coding Agents with an Evidence-Conditioned Execution Layer

Coding agents routinely edit a file or submit a patch before examining enough of the repository to justify the change. ECLoop sits between the agent and the repository and compiles, per task, a list of things the agent must have observed before each kind of edit is permitted; an action whose conditions are unmet gets postponed and the agent is sent back to look. Across all 500 tasks of a real-bug benchmark, two models and two agent frameworks, this lifted first-try success by 4.8 to 11.8 points with no retraining and no change to the agent, and cut tokens by up to 12.1% because the agent stops pursuing edits it cannot support. The detail worth stealing: structured conditions the layer checks beat handing the agent the same requirements as a natural-language summary.

abstract PDF

23.8% → 70.6%

harmful requests refused, when the safety judgment reads the tools as plain text instead of as a schema

Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

A model that refuses a harmful request in chat will often carry it out once it is holding tools, and this paper names an unexpected culprit: the machine-readable schema you declare those tools in. Inspecting the model's internal activations shows schema-formatted tool specifications measurably weaken its refusal signal, so the safety judgment degrades simply because the request arrives next to structured function definitions. The fix separates the two jobs — assess the request against a flattened plain-text rendering of the same tools, then execute with the original schema — lifting refusal of harmful requests from 23.8% to 70.6% and cutting prompt-injection success from 25.6% to 2.5% across four models, with task ability preserved. If you ship tool integrations, that is a strong argument for making the safety call in a separate pass that never sees the schema.

abstract PDF code