A benchmark tests your agent on the day the tools behaved. AgentCheck records a working run, then replays it with exactly one tool response corrupted — a timeout, a stale value, an instruction smuggled into a tool's description — and watches whether the agent flags the problem or sails on and fabricates. Wrapping tool calls in a retry took injected timeouts from surviving 30% of the time to 100%. But a stale value returns a clean 200 OK with no error to catch, so it stays stuck near 3-4 of 10 no matter what you bolt on.
When you evaluate an agent, the tools on the other end of every call behave: they return, they return quickly, and they return the truth. Production is not that. A tool times out, hands back last quarter's number, answers a slightly different question, or hides an instruction in its own metadata — and the failure is silent, because from the model's seat the call simply "worked".
Here are four faults, injected one at a time into a real tool response. Flip between what a naive agent does with each and what handling it correctly looks like.
None of these is a model that "can't do the task". Each is a model handed a tool response that lies, and given no reason to distrust it. A benchmark that only ever serves the truth cannot tell you which of these your agent survives.
AgentCheck injects faults at three points in the loop. Execution faults break the call itself and carry an error signal. Data-quality faults return a perfectly valid response that happens to be wrong — no error, nothing to catch. Security faults smuggle an instruction or a lie through the tool. Filter the library by layer.
The provenance matters for how hard you read the results: 36 execution scenarios come from an existing tool-misuse benchmark and 20 security scenarios from a tool-poisoning one, but the 64 data-quality scenarios were written by the authors — who flag that set as a hypothesis, not settled measurement.
The workbench's third phase is the useful one: re-run the same injected fault with a mitigation wrapped around the agent, and confirm every failed check now passes. Add the mitigations one at a time and watch each fault move — or refuse to.
A retry takes every execution fault to 100% — a timeout or a 500 announces itself, so a wrapper can catch it. The data-quality faults barely twitch: a stale price and a wrong-question answer both come back 200 OK, so there is no signal for the wrapper to trip on. Those need the model itself to doubt a successful call.
Plot each fault by how often the agent survives it with no handling against how often it survives with the best mitigation. Anything on the diagonal is a fault your wrapper could not move. That line is not about difficulty — it is about whether the fault left a signal behind.
It is not that the model is weakest on the hard-sounding faults. It sits safe at the top on prompt injection from the start, and stuck on the line on a stale value — a far more mundane failure — because one arrives as an obvious instruction and the other as an ordinary, successful answer.
Put a number on it. Set your agent's daily tool-call volume and how often a call comes back faulty, then switch on a retry-plus-freshness-plus-relevance guard. The guard clears the faults that carry a signal and misses the ones that don't — so a residue always reaches the user. The blended rates here stand in for the paper's per-fault mix; treat them as illustrative.
The point of the workbench is that this number stops being a guess. You inject the faults, you read the residue, and you can prove a fix moved it before you ship — rather than discovering the gap from a user telling you the price was wrong.