Every tool-augmented model is arbitrating between two fallible sources: the tool's return, and what it already knows. Most evaluations only measure which one it prefers — this one first establishes which one is right, then watches. Across 6,504 episodes built from 542 quality-controlled questions, with each model's own closed-book answer elicited in advance and the tool return's correctness set deliberately, the arbitration is barely an arbitration. Four instruction-tuned models kept a verified-correct answer against a wrong tool return in 6.5% to 17.1% of eligible cases, followed a correct tool in 86.0% to 93.1%, and simply repeated the return in 78.4% to 86.0% of cases where both sources were wrong. And which model looks best flips when nothing changes but the wording of the instruction.
The usual way to study this measures preference: put a tool return in front of a model, see whether it goes with the return or with itself. That tells you nothing useful, because a model that always follows the tool looks identical to a model that follows the tool when the tool is right.
This benchmark closes that hole by fixing both sides in advance. Each model's closed-book answer to each question is elicited and checked first, so its memory has a known truth value for that model. The tool return is then constructed with a known truth value too. Four cases fall out, and only in one of them is following the tool the correct behaviour and disagreeing with it a mistake.
The headline is a gap, not a level. Every model here follows a correct tool between 86% and 93% of the time, which is what you want. The same models keep a verified-correct answer of their own against a wrong return between 6.5% and 17.1% of the time, which is not. Compliance and scepticism are not two ends of one dial in these results — they are separate behaviours, and only one of them has been trained.
The fourth case is the one that should worry anyone shipping a retrieval or tool pipeline. When both sources are wrong, the model does not fall back to caution: it repeats the tool return in 78.4% to 86.0% of cases. Your bad tool output does not get diluted. It gets laundered into a confident answer.
Three instruction wordings were tested with the question text and the tool returns held completely fixed. Retention moved by 7 to 15 points inside a single model, and no ordering of the models survived both the pooled view and every individual wording — the model with the highest retention under one phrasing had the lowest under another.
This is a direct instruction to anyone about to pick a model on a leaderboard number for this behaviour. The number is not a property of the model on its own; it is a property of the model and your exact prompt together. If you change the wording of your system prompt, you have changed this behaviour, and you have no reason to assume in which direction.
Both fine-tuning routes were tried — supervised tuning and preference optimisation — against an asymmetric success criterion the paper is right to insist on: retention must improve, and correct-tool following must not measurably fall. Anything that buys scepticism by making the model ignore good tools has not helped.
Two of four backbones met it under both methods, and the same two failed under both — the outcome tracked the model far more than the training objective. One model gained 41.8 points of retention and lost 27.6 points of tool-following, which is not a fix, it is a mood. And across everything tested, 19 of 20 method-and-model combinations reduced the model's willingness to abstain after a tool error or on an unanswerable input.
Put a number on your own stack. If some share of your tool calls return something wrong — a stale record, a bad row, a retrieval that surfaced the wrong document — the paper's rates tell you roughly what fraction of those reach the user as a confident answer rather than being caught by the model's own knowledge.
Move the sliders. The point is not the exact figure, it is the shape: your model's own correct knowledge is not a safety net behind your tools. It catches roughly one in ten.