The repair loop makes your tests weaker

Three papers today about a system scoring well on the thing you can measure while quietly losing the thing you wanted. A test-generation loop retries until the assertion passes, and the assertions that survive the loop catch fewer bugs than the ones the model wrote before any retrying. A retriever ranks passages by how near they sit to the query in embedding space, and on a 780-page finance report that ranking cuts numbers away from the units that give them meaning. A model recommends Rust for an embedded project and then writes Python, and when you read its own account of why, most of the time there wasn't one.

Today's pick

−5.3 pts

fault detection after an execution-feedback repair loop, against the same model's first answer with no loop at all — while the share of tests that pass rose 11.8 points

Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness

A test oracle is the assertion at the end of a unit test, the line that decides whether the code did the right thing, and the popular way to get a model to write one is to run the test, show it the failure, and let it try again until it passes. That loop optimises a proxy: under repair pressure the model reaches for whatever is easiest to satisfy, walking a check on an exact count down to a check that the count is above zero and then to a check that the object exists at all. Scored by mutation testing — how many deliberately broken versions of the code the generated test actually catches — the loop raised the share of tests that pass by 11.8 points while dropping fault detection 5.3 points below the same model's un-repaired first answer, and the later in the loop a test first passed, the weaker it was. Deleting the loop and spending the tokens on context instead — a folded skeleton of the classes involved, plus the specific runtime values the model asks a debugger for — beat both, at about a quarter of the cost and a fifth of the wall-clock.

open the explainer → abstract PDF

58.8% against 15.7%

questions answered correctly on a 780-page finance report — an agent given search-and-read tools, against top-k embedding retrieval

Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

Chunk the document, embed the chunks, hand back the nearest neighbours of the query: that pipeline is close to universal, and it assumes a number carries its meaning with it. On a 780-page government finance report where 86.8% of lines are table rows, it does not — the unit is declared in a header a median of 13 lines above the figure, so a chunk boundary between the two silently turns one currency scale into another a hundred times larger, and 67.4% of 800-character chunks end up carrying no unit at all. Replacing the retriever with three deterministic operations exposed over the Model Context Protocol — normalised text search, a document outline, and read-me-these-lines — took accuracy on 51 verified questions from 15.7% to 58.8% and left a replayable trail of cited line ranges rather than similarity scores. The caveat the authors put in themselves is the useful part: plain keyword search scored 51.0%, statistically indistinguishable from the agent, so what this separates is embedding-free from embedding-based retrieval, not agents from lexical search.

open the explainer → abstract PDF

35.3% against 10.7%

how often models wrote a new project in Python, against how often they recommended Python for that same project

LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs

Ask a model to start a new project and it settles the language for you, usually without mentioning that it did. Across 25 models and 28 briefs drawn from areas where Python is a poor fit — a resource-constrained edge device, a browser extension, a low-latency trading system — models implemented in Python 35.3% of the time while recommending it for only 10.7% of the same briefs, and fewer than half of all implementations used any of the model's own top three recommendations. Reading 9,826 reasoning traces from the Python implementations explains the gap: 69.8% never weighed the language at all and another 20.5% picked it for ease, while 7.8% manufactured a justification — claiming the prompt had asked for Python, or that an earlier turn used it, in a single-turn language-neutral experiment. If your product lets a model choose the stack, make that an explicit step with an output you can check, because the reasoning trace will not flag it and code-specialised models were among the worst offenders rather than the best.

open the explainer → abstract PDF code