The check has to reach the failure.

Three papers, one shape. A checking tool only improves what it can actually observe: a one-bit does-the-server-start probe rescues nearly every build that would not have launched, and does nothing for a list that scrolls badly. Agents drive twenty chained API calls to the correct final state and then hand back the wrong answer in more than three quarters of their failures. And a model carries a clean, readable no-answer-exists signal that its refusal pathway is not wired to. In each case the capability was there. The path from noticing to acting was not.

Today's pick

86.5% → 99.5%

share of builds that come up at all, with no checking tools versus a single probe that installs dependencies and asks whether the server starts

The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents

Give a coding agent tools to check its own work and the gain lands only where the tool can observe the failure. Holding the agent, the prompt and everything else fixed across 1,116 web applications, six models and eight tool sets, one build in seven never launched with no tools at all; adding one probe that installs dependencies and reports whether the server starts took launches from 86.5% to 99.5%, delivered ten of the twelve points a full shell buys on machine-checked behaviour, and cost less than building blind — 214k median tokens against 262k — because a run that stops guessing re-reads a shorter conversation. The full shell costs 2.35× the no-tools baseline, screenshots add another tenth, and on the one task whose failure is a stutter rather than something visible the screenshots land slightly below the shell. Put the cheap does-it-start check in first, and add each further tool only where it can see the way your thing actually breaks.

open the explainer → abstract PDF code

77% of failures

share of failing twenty-step runs that had driven the system to the correct final state and failed only on the answer they handed back

APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

Chain twenty dependent API calls together and success falls from 92.9% on the individual steps to 74.4% on the whole chain — but not by compounding: the observed rate sits 33 points above the product of the step rates, so steps are not independent failures and models recover mid-chain more than that arithmetic predicts. What actually kills the run is the end of it. Of 219 failing twenty-step runs, 169 had driven the world to the correct final state and failed only the typed answer card — a wrong value, a missing field, an answer never declared. And across nineteen models, best-of-five compresses the whole field into seven points while all-five-of-five spreads it across 44, so if you want a number that predicts production, run each eval case five times and score the worst run rather than the best.

open the explainer → abstract PDF code

0.94 against 0.09

how cleanly one direction inside the model separates answerable from impossible prompts, against how much that direction overlaps the refusal pathway that would act on it

Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

When a model confidently answers a question with no answer — the cotangent of −540°, or calling a string method on the number 1 — it is not that it failed to notice. Across eleven instruction-tuned models from 1.7B to 70B, a single direction in the hidden state separates answerable from structurally impossible prompts with a mean separation score of 0.94, and that direction sits nearly at right angles to the refusal direction that trained safety behaviour runs on, overlapping it by 0.087 on average. Pushing along the recognition direction moves abstention up or down by 33 to 44 points where a matched random direction moves nothing, and the same near-perpendicular geometry is already present in the base checkpoints, so instruction tuning did not create it. The signal is there and nothing routes on it: if you need a model to decline the unanswerable, ask it that question separately or read the signal out yourself, rather than expecting the refusal path to carry it.

open the explainer → abstract PDF