Right value, wrong place

Three papers about the shape of what you hand a model and what it hands back. The first grades structured output twice — did the value come back at all, and did it land at the path it belonged in — and the second number collapses while the first stays high. The second asks whether one agent's message to another should be prose or a typed graph, and shows it is per-task and cheap to decide. The third measures what a thin brief costs an agent.

Today's pick

24% to 74%

of the values a model recalled that it then wrote at the wrong position, once the schema is four levels deep — across six models

Where vs what: structural and content failures in structured outputs

When a model fills a JSON schema you usually check one thing: did it parse and match the shape. This paper grades it twice instead — did the value you asked for come back anywhere, and did it land at the path it belonged in — and the two numbers come apart badly as nesting deepens. At four levels the values are nearly all present (91% to 96% for the stronger models) while 24% to 74% of them sit at the wrong position, so a single accuracy score reads an addressing problem as a content problem. Repeated key names and opaque field names each add about eight points to the misplacement rate, making a schema rename the cheapest fix available; training a small model on a reward that pays for correct placement took it from 26% to 63%.

open the explainer → abstract PDF

+12.7 pts

more customer-service tasks completed when a 155-token classifier hands the next agent a typed dependency graph instead of prose, on messages 3.2 times smaller

Routed graph handoff: picking the delegation format per task

Agents delegate to each other in prose, and that prose is a large share of what you pay for. Replacing it with a typed dependency graph — goals, entities, tool calls, and edges saying which step depends on which — shrinks a customer-service handoff from 730 tokens to 226 and lifts task success by 12.7 points, but the same format loses 14.6 points on a benchmark full of tasks that need the agent to iterate or interpret free text, because a graph node cannot say "be flexible". A 155-token classifier picking graph or prose per delegation keeps the wins and erases the losses, and only works if the receiving agent's prompt explains how to read a graph. Route your deterministic ordered lookups through a structured format and leave everything else in prose.

open the explainer → abstract PDF

+29.7%

more tokens burned per coding task when the brief is cut to a bare user story instead of a full eight-section specification, across 2,700 runs

Can your AI agent be cheaper? Task specifications and token spend

Two engineers hand a coding agent the same bug and get bills differing by more than the model choice does. Across 2,700 runs on five real repository bugs, cutting a full eight-section specification down to a bare user story raised token spend by 29.7% and turns by 16.4% while the solve rate barely moved — the agent spends the tokens you saved by not writing the brief, and how much it spends varies by task from 13% to 115%. Run-to-run spread is stable at a median 1.34 times and no prompt change moved it, so the brief is a real lever rather than noise. One probe run costing 11 US cents prices the whole grid of brief styles and efforts for an unseen task to a median 36% error, against 161% with no probe.

open the explainer → abstract PDF