507 business workflows — refunds, booking changes, claim intake, account disputes, staff provisioning — run in a sandbox where the agent talks to a simulated customer through isolated tool servers, and every attempt is graded by executing checks against the backend afterwards. The strongest model finishes 65.4% of them on one attempt and 91.1% if allowed twenty. Required to finish on all twenty, it manages 25.3%. Most of the failures end cleanly, with valid tool calls and a confident closing message.
Two numbers can be computed from the same twenty runs. The first asks whether any run succeeded — useful if a human is going to pick the good one out of a pile. The second asks whether they all did — which is the question a customer-facing agent is actually answering, because nobody re-rolls a refund. Drag the number of attempts and watch the two diverge.
The mixture behind the curve is fitted to the paper's three measured points and says something blunt about the task set: roughly 31% of these workflows the strongest model does essentially every time, roughly 60% it does about three times in five, and roughly 9% it never does at all. The middle band is the entire product problem.
Five domains, all built the same way: a backend with real tables, tools exposed over the same protocol an agent would use in production, a written policy, and a simulated customer who knows what they want but not what the agent needs. Retail is the friendly one — few entities, short chains. Consulting IT and HR provisioning is eighteen backend systems and thirty tables. Step through the domains and watch the whole model ranking compress or spread.
Two things fall out of the table. The rank order of models barely changes from domain to domain, so this is not measuring a specialism — auto insurance and consulting provisioning are simply harder work. And every model swings between 18 and 48 points across the five domains, which means a domain that is hard is hard for everyone: swapping in a stronger model raises the floor, it does not rescue the workflow.
Each task carries executable checks that run after the conversation ends and look at the terminal state of the backend: the right rows changed, nothing else touched, nothing missing. 477 of the 507 tasks are graded on state alone; 30 add a check on the final response, for things like a required disclosure or a confidentiality rule. Step through four attempts on the same refund task and watch the verdict flip.
This is why the benchmark can say something a response-level score cannot: three of those four attempts terminate cleanly, and two of them make valid, well-formed, state-changing tool calls. Only one is right. Clean termination is not a proxy for task completion, and if your eval reads the final message you are measuring the wrong thing.
The paper labels every failed trial with its dominant signature. The result is lopsided: the overwhelming majority are tool-usage failures — a precondition not met, a lookup that came back empty, a call that errored and was never retried differently. The rest divide into changes made to the wrong entity, conversations closed without resolution, and the rare case where the agent retrieves everything correctly and then never writes anything at all. Step through the signatures.
The distribution is not flat across models: on the strongest model 89.6% of failures are tool-usage errors, while on another model in the suite wrong-entity writes account for 27.8% of failures. The first shape fails loudly and recovers if you retry; the second writes something plausible to the wrong record and does not.
Take the metric to your own agent. Set the per-attempt success rate you measure today, the number of tickets a day it handles, and how many of them a customer would notice going wrong. The all-attempts column is the paper's metric applied to your traffic: it is the share of task types your agent handles every time, not just the share of attempts that pass.
The lesson to steal is the reporting line, not the number: run your twenty hardest task types twenty times each and publish the all-twenty figure next to the single-attempt one. It costs an afternoon of inference and it is the only number that matches what a repeat customer experiences.