You cut the tokens. The bill went up.

The industry spent this week on the two things that quietly bleed an agent: security and cost — sandboxing agents, red-teaming prompt injection, watching the meter. Today's papers sit on that same nerve, the places an agent costs you money, trust, or a working fix without ever raising an error. The pick is the one that stings: the context-trimming you did to save money made it go up.

Today's pick

−38% tokens, +6.8% bill

cutting 38% of an agent's tool-output tokens paired with a 6.8% higher bill (95% CI +2.8–11.3%) — because cached context, not fresh tokens, is ~80% of the cost

Token Reduction Is Not Cost Reduction

The intuition that trimming an agent's context saves money is so obvious almost nobody checks it — this team checked, across 2,848 billed runs and about $643 of real API spend, and it is wrong. An arm that removed 38% of tool-output tokens paid 6.8% more, not less, because roughly 80% of the bill is not fresh tokens at all but prompt-cache traffic: re-reading the cached conversation prefix on every model call, which scales with how many turns the run takes. Worse, aggressive compression corrupted the verbatim anchors an agent needs to apply an edit, so it searched again, added turns, and re-transmitted the whole prefix — turning a nominal saving into a higher bill and fewer solved tasks. Measure cost per successful task rather than tokens, and spend the effort on grounding the context precisely at the right lines instead of shrinking it.

open the explainer → abstract PDF

~37% don't exist

of the skills an agent recommended installing exist in no registry — and the fake names repeat across runs and models, so an attacker can pre-register them

Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents

Ask an LLM agent which skill or plugin to install for a task and roughly one recommendation in three names something that exists in no registry at all — 36.9% for agents, and 43.1% on real developer questions across 15,000 prompts. The dangerous part is that these fabrications are not random noise: agents produce the same 5,669 fake names again and again across different prompts and models, which hands an attacker a reliable list of names to register malicious packages under and wait for an agent to install one. The obvious defence — grounding recommendations in a real catalog by retrieval — cut the hallucination rate from 40.8% to 3.2% but also crippled usefulness, leaving even the best-defended system suggesting the right skill only about one time in six. If your agent installs skills or tools from an open catalog, treat every recommended name as unverified until it resolves against a registry you control, and don't assume grounding is a free fix.

abstract PDF

up to 85% of verdicts flip

share of a judge's correct/incorrect decisions that reversed once a reference answer was added to the prompt — reference-free judges systematically over-credit wrong answers

LLM Judges Can Be Too Generous When There Is No Reference Answer

If you score model outputs with an LLM judge and no gold answer in the prompt — the usual setup for open-ended tasks — the judge leans generous, waving through answers that are actually wrong. This paper measures how fragile that is: adding the reference answer back into the prompt flipped the judge's correct/incorrect verdicts by as much as 85% in some settings, meaning a large share of your reference-free passes would have been failures with the truth in view. The authors' fix is to calibrate first — run a sample with reference-aware judging to measure how well the judge actually knows the task before you trust it to grade blind. Before you lean on a reference-free judge in an eval or a reward signal, calibrate it against reference-aware scores on a labelled slice, or you are measuring the judge's generosity rather than your model's quality.

abstract PDF

200–5000× faster

than prompting a model to find the culprit step in a failed agent run — and it learns from successful runs alone, with no error labels

Tracing Agentic Failure from the Flow of Success

When a multi-step agent fails, working out which step actually broke it usually means paying another model to read the whole trajectory and guess — slow, and it needs examples of failures you may not have collected. This method learns the shape of successful runs alone — no labelled error steps — by modelling how a good trajectory flows through the model's latent space, then flags the step where a failing run departs from that flow as the anomaly. Trained on just 100 successful trajectories it beat prompting-based attribution by about 20 points of F1 in-domain and 7 out of distribution, while running 200 to 5000 times faster. If you run agents in production and mostly have logs of the runs that worked, this is a cheap way to point at the likely culprit step instead of re-reading every failing trace by hand.

abstract PDF

ECE down up to 54%

better match between a step's stated confidence and whether it actually worked, versus the best training-free baseline — with no training and no step labels

Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

Holding back before a risky or irreversible tool call — a payment, a delete, a send — needs a trustworthy confidence signal for the step the agent is about to take, and you need it before the step runs. This approach keeps a memory bank of past steps labelled in hindsight as productive or not, and when a similar step recurs it retrieves those experiences into the critic's prompt, all without training or ground-truth step labels. That improved calibration — how well the stated confidence matches whether the step actually succeeds — by up to 54% over the strongest training-free baseline across three agent benchmarks and three critic models. If you want a gate in front of an expensive action, a retrieval-fed critic gives you a calibrated number to threshold on rather than the model's untrustworthy raw certainty.

abstract PDF

0.45 → 0.58 faithful

citation faithfulness rose as the generator saw more of each source, but coverage stayed pinned near 0.22 — that ceiling is set by retrieval recall, not exposure

On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage

Two different things go wrong when an agent cites its sources, and this paper shows they need different cures. How faithful each cited claim is — whether the source actually supports it — is bounded by exposure, meaning how much of each document the generator gets to see: widening that from 400 to 1500 characters lifted faithfulness from 0.45 to 0.58 for about 235 extra output tokens. But whether the right sources get cited at all — coverage — sat stuck near 0.22 at any exposure, because it is capped by retrieval recall, which was pinned around 0.40. So if your grounded answers are unsupported, feed the model more of each retrieved document; if they miss whole sources entirely, no amount of context helps and you have to fix retrieval instead.

abstract PDF