Some requests have no valid answer. The package was never published. The compressor would have to shrink every input, which counting forbids. The store would have to stay consistent and available through a partition. The correct response is to say so. Across twelve open-weight code and reasoning models, 270 such prompts in six languages and 4,332 judged responses, the models instead produced ungrounded code on about 60% of them and refused on 27%. The control tells you what kind of failure this is: each impossible prompt has a near-identical solvable twin — one crate name swapped, one constant relaxed — and across all 91 of those, models wrongly refused zero times. Nothing is too cautious here. The scepticism is simply absent.
The suite is built the way a good experiment is: in pairs. Each adversarial prompt asks for something that cannot exist, and each is shadowed by a control that is a minimal edit of it — the nonexistent Rust crate rand_constant_time replaced by the real subtle, the impossible guarantee relaxed to a possible one. Same phrasing, same task shape, same length. The only thing that changes is whether an answer exists.
That pairing is what makes the headline number mean something. A model that refused everything would score well on the adversarial half and catastrophically on the controls. No model does. Over-refusal on the 91 controls is flat zero for all twelve — not low, zero. Whatever is going wrong, it is not an over-tuned safety reflex; it is that nothing in the model's response pipeline is checking whether the thing being asked for is real.
The rate is also wildly uneven across kinds of impossible. Made-up package names are near-total failures. Prompts that violate a proven theorem are much better handled. The gap matters because it tells you which impossible things your own product is likely to hit.
Read the two ends of that list together. A prompt naming an npm package that does not exist gets fabricated code 98% of the time. A prompt that is a lightly modified version of a classic textbook problem — the shape most likely to be memorised — gets fabricated 8% of the time. The model is not failing at reasoning. It is failing at the specific act of doubting a name it has been handed.
The taxonomy splits impossibility along its first axis, groundedness, into two branches. Absolute impossibilities violate something universally true — a halting-problem decider, a compressor that shrinks every input, a polynomial-time 1.5-approximation for minimum vertex cover, a store that beats the CAP theorem. Relative impossibilities violate a contingent fact about the world as it currently is — this package was never published, this compiler flag was never added, this protocol does not exist, this language has no such feature.
The distinction earns its keep the moment you compare weak models with strong ones. Line up the four models with the lowest fabrication rate against the four with the highest, and the improvement is almost entirely on one branch.
Theory falls from 82% to 23% as models get better. Plausible-sounding entities barely move: 98% to 87%. Being asked for a smarter algorithm than mathematics allows is something a stronger model can catch. Being handed a package name that sounds exactly like a hundred real package names is not, because there is nothing about the name to catch. That asymmetry is the practical core of the paper — the failure that survives scaling is the one your product is most likely to trip.
The spread across models is enormous — 26% fabrication at the best, 90% at the worst — and it tracks recency more than size. The 7B models from an earlier generation almost never refuse; newer code models refuse roughly half the time. But the ordering has one entry that breaks the pattern badly enough to be worth its own control.
Now the control. Take two models of the same size from closely related base families, one a reasoning model and one not, and put them side by side.
The reasoning model is the worse one, by a wide margin, on both numbers. Extended deliberation about how to satisfy the request is not the same capability as noticing the request cannot be satisfied — and on this suite the deliberation appears to work against the noticing, because it is deliberation aimed at producing an answer.
Every prompt in the suite is wrapped in one of ten natural framings — the sort of thing that varies for free in production. The same impossible request, presented as routine work, gets fabricated code far more often than when it is presented as a question.
Seventeen points, from a sentence that carries no technical content at all. Regress the outcome on prompt identity and on model identity separately and the same thing shows up in the variance.
The last axis of the taxonomy is what the model does when it fails, and it is where the trajectory of the field shows up most clearly. Two thirds of all failures are confident fabrication: code delivered with no hedge at all. A fifth are hedged compliance — the model flags a doubt in a comment and then writes the code anyway.
Hedged compliance rises from 4% of failures on the oldest models to 43% on the best. That is the shape of a capability arriving in two stages: models learn to notice that a request is suspect well before they learn to act on the noticing. Which is worth knowing when you read your own logs — a comment saying "note: this API may not exist" above a confident import is not a near-miss, it is the current state of the art.
Every coding eval you already run scores whether the agent produced a working answer. None of them score whether it should have. The cheap fix is a canary set: a small number of requests in your own domain that cannot be satisfied, scored only on whether the agent says so.
Sizing it is arithmetic, not judgement. At the paper's measured rates, a handful of canaries is already enough to make the failure visible on every run rather than once a quarter in an incident.
The rate slider spans the paper's real range, from its best model to its worst, so you can see how much of this is a model-selection problem: at the top of the range a suite of twelve canaries catches something on essentially every run; at the bottom it still catches something nine runs in ten. Either way the number you want on the dashboard is the refusal rate, and today almost nobody measures it. The scenarios above are illustrative — the paper evaluates models, not release processes — but the rates driving them are the paper's own.