The tool lied and the agent believed it

Three papers today about the gap between an agent finishing and an agent being right. The first handles tool results that are wrong but well-formed — a cached error page, a negative price — and nearly triples completion by telling the agent which property broke and which other tools could get the value. The second grades 507 business workflows on the state of the backend afterwards, and finds the strongest model completes 65% of them once and 25% of them twenty times out of twenty. The third compiles the written policy into a graph a separate checker walks at every user turn.

Today's pick

10.9% → 28.1%

of fault-injected tool workflows the agent completed once a note in the tool result named the broken property and listed other tools that could get the value — four models, two provider families

Outcome Monitors: Recovery Affordances for Silent Tool Failures

When a tool call times out the agent sees the failure and routes around it; when the call returns a cached error page, a stale record or a negative price in exactly the expected shape, the agent eats it as fact and confidently finishes the wrong task. The authors mine "outcome contracts" — invariants like the argument you passed echoing back in the result, prices staying positive, statuses coming from a fixed set, totals still adding up — from traces of unrelated tasks or straight from public API schemas, then run them as a monitor that never blocks anything: on a violation it passes the raw result through untouched and appends a short receipt naming the broken property and listing public tools that could get the value another way. Completion under injected faults goes from 10.9% to 28.1% across four models in two families, replicates in a third, and adds 14 and 12 points on a retail customer-service benchmark — and the control that matters is stripping the list of alternative tools out of the receipt, which erases the entire gain, while vaguer wording or later delivery changes nothing measurable. Detecting the bad value is not what pays; naming what else the agent can call, inside the tool result, at the moment the value turned out to be junk, is.

open the explainer → abstract PDF

65% vs 25%

of 507 business workflows the strongest model completed on a single attempt, versus completed on all twenty attempts — same tasks, graded on the final state of the backend rather than on what the agent said

One Success Isn't Reliability: a sandbox and benchmark for agents in stateful business workflows

Scoring one attempt tells you a successful path exists, not that your users will land on it. This benchmark puts an agent, a simulated customer and isolated tool servers in a sandbox across 507 retail, travel, insurance, banking and IT-support workflows, then grades each attempt by running checks against the backend afterwards — the right records changed, nothing else touched, nothing missing. The strongest model scores 65.4% on one attempt and 91.1% if allowed twenty, but only 25.3% when it has to succeed on all twenty, and most of the failures terminate cleanly and look successful, with roughly 12% writing a valid change to the wrong entity, date or amount. Borrow the metric before the benchmark: run your top twenty agent tasks twenty times each and report the all-twenty number, because that is the one your support queue is measuring.

open the explainer → abstract PDF code

0.42 → 0.62

average share of customer-service tasks passed on all four attempts across airline, retail and telecom, after the written policy was compiled into a workflow graph an external checker walks at every user turn

PolicyGuide: from guarding one action to guiding the whole workflow

Agents break policy in two ways and the usual guardrail catches one: it blocks the forbidden action, but cannot see that the agent skipped verifying who it was talking to, or never confirmed before charging the card. Here the written policy is compiled offline into a graph of typed nodes — read-only calls, mutating calls needing authorization, required confirmations, decision branches — and a separate verifier runs at each user-turn boundary, walking from the position it stored last turn to the first step the conversation does not yet satisfy, then injecting that step as the agent's next instruction. Mean pass-on-all-four-attempts goes 0.42 → 0.62, and the domain whose steps have a mandatory order goes 0.25 → 0.675; putting the same graph in the agent's own system prompt gets 0.325 there, so the win is an external checker holding position across turns, not the agent knowing the procedure. Keep procedure state in code beside the conversation rather than in the model's head, and budget roughly forty cents a conversation for the checker.

open the explainer → abstract PDF