A tool call can go wrong two ways — the model picks the wrong tool, or it picks the right one and fills the arguments in wrong — and either mistake quietly poisons every step that reads its output. This study measures the same calls twice: once with every earlier step handed to the model already correct, and once inside the trajectory the model actually produced. The difference between those two numbers is error propagation with nothing else mixed in. By step six it has eaten roughly 70% of what the model could do on a clean history. Then comes the part that should change how you run evals: graded against one fixed correct trajectory, recovery is not merely rare, it is unobservable — 0 of 580 corrupted steps ever came back on track, because after the first divergence the "right" answer depends on tool outputs the model was never shown.
A tool call is correct when it does two separate things right: selects the tool the step needs, and supplies arguments that are actually right. Split them apart and they fail differently. A malformed argument that breaks the schema comes straight back as an error, and the agent gets a chance to notice. An argument that parses cleanly but carries the wrong value executes silently, returns something plausible, and goes into the context as fact.
That second kind is the one that compounds. Every later step reads what the earlier ones produced, so one quiet wrong value at step two is the input to steps three through eight. The study's whole design exists to separate that spreading from the model simply finding deeper tasks harder.
The design is a pair of parallel arms. In the first, the model is handed a correct history at every step and asked only to produce the next call — this measures what it can do when nothing upstream is broken. In the second, it runs its own trajectory, so the inputs to step five are whatever step four actually produced. The first number is capability; the second is capability minus everything its own past cost it.
The ratio between them is the loss, and it is steep. On one model it goes from nothing at step one to 0.145 at step two, 0.391 at step four, and 0.686 at step six. A second model lands at 0.684 at the same depth from a completely different starting accuracy — which is the point. Two models with different capability lose roughly the same proportion of it by step six.
Here is the finding that outlives the numbers. Almost everyone scores tool trajectories the same way: fix one correct sequence of calls in advance and mark each step right if it matches. That works perfectly until the model diverges — and then it stops measuring the model at all.
After a wrong step, the correct next call depends on the value the tool would have returned had the previous step been right. That value is generated from tool constants the model was never shown. It is not that the model fails to recover; it is that recovering is not derivable from anything in its context. Zero of 869 corrupted steps scored correct, and zero of 580 returned to the gold path against an expected 0.0058 by chance.
The consequence is that a model fitted to this data reports confident numbers for a quantity the scoring rule already decided. The fit returns 0.92 for a severity that is exactly 1.000 by construction, and 0.73 for a recovery rate that cannot be observed at all.
Step the chain below. A single silent argument error at the step you choose is enough to take every call after it off the gold path — not because the model gets worse, but because it is now answering a different question correctly.
Flip the scoring rule to see the difference the paper is arguing for. Exact match calls everything after the divergence wrong, forever, and tells you nothing about whether the model is still reasoning properly. Conditional scoring asks the only question that has an answer: given the value the model is actually holding, did it apply the rule correctly?
Suppose your tool-calling agent gets each step right with some fixed probability on a clean history. That number is the one your evals report, because your evals almost certainly hand each step its correct inputs. The number you actually ship on is the one that survives to the end of the chain.
Move the two sliders. The left bar is what your eval reports, and it barely moves with depth — because an eval that hands each step its correct inputs is measuring a clean history every time. The right bar is the same call at the same depth inside the model's own trajectory, after the paper's measured loss. The gap between them is what your eval is not telling you.