Most products bill step-by-step reasoning on every request and treat it as a quality setting. Across three open models and five benchmarks, pre-registered, it recovered 54 to 68 points on problems that need many dependent steps and 0 to 4.6 points on problems that are one lookup deep. Within a single benchmark the same gradient appears: answering straight out falls from 45.5% to 15.4% as the number of chained steps in a problem rises, while reasoning out loud stays flat near 85%. It is not an enhancer. It is a way to buy serial steps the forward pass cannot fit.
Every point below is one model on one benchmark: how it scores answering directly on the horizontal axis, how it scores reasoning out loud on the vertical. The dashed line is where reasoning changes nothing. If reasoning were a general upgrade the points would sit in a band above that line. They do not — they split into two groups with almost nothing in between, and one point falls a long way below it.
The split is not subtle and it is not about difficulty. On the shallow benchmarks the direct-answer baseline is already at 68.8% to 94.5%, so there is room to improve and reasoning does not take it. On the deep ones the same models sit as low as 4.7% without reasoning and above 63% with it. Same models, same day, opposite verdicts.
Benchmark averages hide the mechanism, so the paper scores every item for serial depth — how many results have to be computed in order, each waiting on the one before. On word problems that is the count of calculator steps; on competition maths, the number of intermediate equations; on code, how deeply the solution nests. Then it splits a single benchmark by that depth and re-scores. Step through the bins.
This is the result that makes the framing more than a slogan. A benchmark-level correlation between depth and recovery would be suggestive; a within-benchmark gradient on the same items, scored the same way, means the thing being measured is a property of the problem rather than of the dataset. Pooled across all fifteen cells, the depth-to-recovery rank correlation is 0.661 at p = 0.007.
Group the fifteen cells by whether the benchmark's items are deep or one lookup deep and the picture is clean. Flip the switch and read the deltas — the same three models, the same day, the same decoding settings.
One honest caveat, and the author raises it first: the direct-answer baselines on the shallow benchmarks run from 68.8% to 94.5%, high enough that a ceiling — or contamination from pretraining — could be hiding an effect. So the shallow null is a good reason to stop paying for reasoning on multiple-choice work, and a weak test of the architectural claim behind it. Those are different things and the paper keeps them apart.
Code generation sits between the two groups, and it is the only benchmark where model size changes the sign of the effect. Step through the three models on the same task.
The paper does not explain the 28.7-point drop, and says so — it calls it genuine but unexplained and notes the comparison is not clean, because the direct-answer arm on this benchmark was allowed 256 output tokens rather than the 32 the multiple-choice arms got, so it was never a true single-pass condition. What survives regardless is the shape: on the task where the two arms are closest in capability, the smallest model is the one that reasoning hurts.
Take your own traffic and split it by whether the request needs chained steps. On the shallow share, the measured recovery is between 0.0 and 4.6 points — and every one of those requests is still generating a reasoning trace. The ceiling below is the paper's own: a 2,048-token budget for reasoning against 32 for a direct answer.
The routing rule this suggests is cheap to build and does not need a classifier. Ask what the request needs computed in order: a support reply grounded in one retrieved passage is one lookup; a multi-hop question, a schedule with constraints, a refactor across three files is not. Send the second group to a reasoning model and let the first answer directly — and measure both, because the null on shallow work is the paper's weakest result, not its strongest.