Chain twenty dependent API calls together and success falls from 92.9% on the individual steps to 74.4% on the whole chain. The interesting part is where the loss is not. It is not compounding — the measured rate sits 33 points above what independent step failures would predict, so agents recover mid-chain far more than the arithmetic says. And it is mostly not in the work: of 219 failing twenty-step runs, 169 had driven the system to exactly the right final state and then failed only on the answer they handed back. Meanwhile best-of-five squeezes all 19 models into seven points while all-five-of-five spreads them across 44, so the scoring rule decides whether your eval can tell these models apart at all.
The benchmark grows its API worlds forward, one subtask at a time: each subtask is proposed against the state the previous one left behind, admitted only after an automated self-test verifies its grader and an oracle establishes that it is solvable, and tied to its predecessors by a ledger that forces real dependencies so a chain cannot quietly become twenty independent tasks.
Success falls with chain length, but the descent is not smooth and it is not the product of the step rates. For one frontier model, 94.9% per subtask predicts about 35% over twenty steps if each step failed on its own; it actually scored 72.7%. Pooled across the panel, the measured rate sits 33 points above the independence prediction.
That gap is worth internalising before you design a retry policy. Agents do recover from bad steps mid-chain, so any model of your workflow that multiplies per-step reliability will badly under-predict what you see. The failures that remain are concentrated somewhere else entirely.
Instead of one end-to-end bit, every task is scored against seven engineering capabilities, so a failure can be attributed rather than merely counted. Pick one and see where it sits.
Carrying state across calls is the strongest at 91.2%. Choosing between similar-looking endpoints is the weakest at 76.1%, and refreshing a token mid-session is second weakest at 80.7%. Those two are worth pausing on, because they are the ones your integration owns rather than the model: discovery gets easier when the tool surface you expose is small and unambiguous, and authentication failures come from credentials expiring inside a long run, which is a property of your session handling.
Grading has two surfaces and they answer different questions. The state check traces a distinctive value minted inside the mock through the API data flow into the workspace, which proves the required calls actually ran. The answer card compares the final response to the reference field by field with typed comparators — numbers as numbers, currency as currency, sets as sets.
Of 219 failing runs on clean twenty-subtask chains, 212 submitted an answer at all, so abstention is rare at 3.2%. 180 failed the answer card. And 169 of those — 77% of every failure — had every state check pass. The agent drove the world to exactly the right place and then returned a wrong value, a missing field, or an answer it never declared.
The paper is careful that the finding is concentrated: two of the worlds hold 167 of those 169 runs. But the median failing run made 15 tool calls before losing the thread, which is the detail that should change what you instrument. Grade only the final string and you cannot see this class at all; check only side effects and you will score these runs as successes.
Run each task five times and you can score it three ways, and the three disagree about what the panel even looks like.
Which of those predicts production depends on whether your user retries. For a workflow that touches real state — a payment, a ticket, a booking — the run that half-works is not a free retry, so the all-five number is the honest one. It is also the only one that separates the field, which makes it the number worth optimising.
The mechanics transfer directly to whatever eval suite you already have. Pick the single-run pass rate you believe your agent has on a case, and compare what a best-of-five report would say against what all-five-of-five would say. The gap is the part of your reliability story that a one-shot eval is hiding.
The second lever is what you grade. Score only the returned answer and the 77% finding above is invisible to you as a class — you will see failures, conclude the agent cannot do the work, and rewrite a prompt that was never the problem.