Tool-use work has gone almost entirely into picking the right tool and ordering the calls. Filling in the arguments was left to hope — and on real cloud-network APIs even frontier models get fewer than half of them exactly right, one scoring 29.4% with no examples. The finding worth borrowing is that the model is already representing its own error: a logistic regression on the hidden state captured just before it writes each parameter value predicts that value's correctness at 0.986 AUC, against 0.914 from the token probabilities. Use that score to filter self-generated training data and to rerank sampled candidates, and average exact match moves from 19.7% to 59.6% — of which ordinary fine-tuning is 51.6 and the probe is the last 8.0.
An agent that picks the right API and then passes it a malformed nested object has failed just as completely as one that picked the wrong API — and the failure is harder to see, because the trace looks correct right up until the call returns an error. The benchmark here is built from real cloud-network APIs and grades every instance on four measurable properties of the arguments: how deeply the required fields nest, how many fields depend on the value of a sibling, how many values must be carried over from an earlier call's output, and how deeply buried those carried values are. Step through the levels.
Here is the observation the rest of the paper is built on. Take the hidden state at the moment just before the model emits a parameter value — one vector, from one layer, about two-thirds of the way up the network — and fit a plain logistic regression on it to predict whether that value will turn out to be correct. That classifier reaches 0.986 area under the ROC curve in-domain, comfortably past the 0.914 you get from the model's own token probabilities. Switch between the settings and watch how far that signal carries.
The shape of that result is the whole practical story. Within one model, one domain and one sampling temperature the probe is close to an oracle, and it survives a checkpoint update at 0.982 — so you can fit it once and keep it through a fine-tuning round. Point it at a different dataset and it falls to 0.711–0.770, which is still above chance and nowhere near usable as a gate. This is a per-domain instrument, not a general-purpose confidence score.
Once you have a reliable per-field correctness estimate, there are exactly two places to spend it: before training, to decide which self-generated examples are worth learning from, and at inference, to decide which of several sampled candidates to actually send. The paper builds both. The training-time use is a bootstrap loop; the inference-time use is a reranker with three strategy families, and the interesting one is not the obvious one. Step through the loop first.
At inference the model samples a pool — one greedy call plus eight to twelve sampled ones — and the probe scores every parameter in every candidate. The naive move is to rank whole calls and send the best. The better move is to stop treating a call as atomic: take each field from whichever candidate scored highest on that field and rebuild the call from parts. Try all three strategies on the same pool. illustrative pool
The headline is a move from 19.7% to 59.6% average exact match, and it deserves an honest decomposition, because most of that is not the probe. Fine-tuning a model on a labelled set of correct calls — the thing you would have done anyway — carries you from 19.7 to 51.6. Filtering the self-generated data by probe score adds 8.0 on top of that, and reranking at inference adds roughly 4 to 5 more. Switch models and watch the arms.
You can apply the difficulty rule to your own tools this afternoon, and it is worth doing even if you never build a probe — because it tells you which of your tools are in the band where frontier models get fewer than half the calls right. Set the four properties of your hardest tool schema and read off the level. The rule is the paper's, exactly; what is illustrative is treating your schema as comparable to theirs.
And the gate that decides whether the rest of the paper is available to you at all: the probe reads hidden states, which no hosted API exposes. If you serve your own weights this is a weekend's work with a real payoff. If you do not, the transferable part is the difficulty grading and the reranking shape — sample a pool, score it with whatever signal you can get, and rebuild the call field by field rather than picking one whole candidate.