The model was never the variable.

Seven papers, one shape under all of them: the thing that moved the number was never the model. A guardrail says yes and the world moves before the action lands. Go six steps deeper into a tool chain and 70% of the capability is gone. Reorder the same candidates and a third of what a scorer keeps changes. Compress the context and the language it was trained on keeps its meaning while the others quietly lose theirs. Every one of these lives in the part you own.

Today's pick

3.4% – 24.7%

share of approvals that were genuinely correct when checked and no longer admissible by the time the action ran, across five systems

Approved Too Late: Verdict Staleness in Guarded Self-Adaptive Systems

A guardrail checks a proposed action against the current state and says yes. Then time passes — the verdict is computed, delivered, the action queued — and the state it approved against is not the state the action lands in. Replaying logged decisions eight steps later across five self-adaptive systems, the reference verdict had flipped on 5.3% to 48.4% of candidates, and of the approvals that were genuinely correct at check time, 3.4% to 24.7% were inadmissible by the time they ran. The fix needs no model of the system: estimate how long an approval stays good from its safety margin divided by how fast the relevant signal has lately been moving, and refuse to act past that horizon. It cut expiry to between 0% and 1.8% in all five, and every audited judge stream had a nonzero invalid rate at use time.

open the explainer → abstract PDF

70% gone by step six

share of a model's clean-history tool-calling ability lost to its own earlier mistakes, six steps into a chain

Invocation-Level Reliability of Tool-Using Agents

Score a model's tool calls twice — once with every earlier step handed to it correct, once inside its own running trajectory — and the gap between them is error propagation on its own. Six steps in, roughly 70% of what the model could do on a clean history is gone. Worse, graded against one fixed correct trajectory recovery cannot be seen at all: after the first wrong step the right answer depends on tool outputs the model was never shown, and 0 of 580 corrupted steps came back on track. Grade each call against the state the model actually holds instead.

open the explainer → abstract PDF

6.5–17.1%

how often a model kept its own verified-correct answer when the tool handed it a wrong one — against 86–93% compliance when the tool was right

MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models

When a tool return contradicts what the model already knows, one of them has to win, and it is almost always the tool. Across four instruction-tuned 7-9B models and 6,504 episodes where the truth of both sources was known in advance, models kept a verified-correct answer against a wrong return in only 6.5% to 17.1% of cases, followed a correct tool 86% to 93% of the time, and echoed the return in 78% to 86% of cases where both were wrong. Fine-tuning repairs retention on some backbones, but 19 of 20 combinations made the model less willing to abstain.

open the explainer → abstract PDF

54 successes in a row

a stuck agent loop no error-rate circuit breaker could see, because every call in it returned success — it stopped only when a person killed the run

Agent Mesh: Reliability Primitives for Non-Idempotent Delegation

Retry, timeout and error-rate circuit breaking were designed for stateless service calls, and all three assumptions under them break once the unit of work is a delegation the agent cannot un-do. A study of 147 numbered incidents across 81 runs of one production platform shows the shape: the same call issued 54 times in eleven minutes, every one returning success and invisible to any error-rate breaker; a progress signal computed over the failing check's name, constant by construction, declaring a healthy run stalled on its third repair round and taking it from six working components to three; and twelve incidents where the safety layer blocked correct work, the worst costing 107 agent turns and zero accepted writes.

open the explainer → abstract PDF

+23.4 to −9 points

what the same manager-and-worker scaffold did to nine different models on the same 100 problems

Zero-Shot Self-Orchestration with Ledger-Based Control

A manager that writes a plan, curates a task list and decides when to stop, with workers reading and writing a shared filesystem workspace, is the shape most agent frameworks push you toward — this measures what it actually buys across nine models on 100 competitive-programming problems. The answer is conditional and worth knowing before you build one: one model gained 23.4 points, two gained 8 to 10.6, and one lost 9 because the planning stage talked itself out of a correct approach and into a slower one. About a fifth of the largest gain was simply the scaffold rescuing runs that had blown their context. It roughly triples the token bill and still beats moving up a tier — one setup reached 85.0 against a frontier model's 87.4 at a fifth of the cost per pass.

open the explainer → abstract PDF

57–62% vs below zero

usable value left in the context after keeping a third of the tokens — English against Chinese, where the compressed context scored worse than sending none

Lost in Compression: A Cross-Lingual Audit of Prompt Compressors

Extractive compression drops the low-information tokens from your context before you send it, and the learned compressors everyone reaches for were trained on English. Audited across ten languages and five scripts against both a no-context and a full-context anchor, keeping a third of the tokens left English holding 57–62% of the context's usable value, Lithuanian 10–24%, and Chinese below the no-context baseline — worse than sending nothing at all. The cause is the supervision language rather than the architecture: a multilingually trained compressor showed no gap and the deterministic baselines showed none either. Nothing raises an error when this happens; it shows up only as a gap between the compression rate you asked for and the one you got.

open the explainer → abstract PDF

0.010 apart, 0.66 agreement

five scorers within 0.010 of each other on ranking quality whose kept sets overlapped only 0.66 to 0.84 when the same candidates were shown in a different order

Equal Ranking Quality, Different Decisions: Order-Consistent Scorers

Rerankers, reward models and multi-document scorers grade candidates inside one prompt, so every score depends on the order they arrived in — and the ranking metric you select on hides that completely. Five trained scorers within 0.010 of each other on ranking quality kept sets overlapping only 0.66 to 0.84 across reorderings, and a purpose-built commercial reranker overlapped 0.667 with itself. No prompt-side fix tested removed it; the one that did lift ranking quality left all three downstream decisions exactly where they were. Report what your threshold keeps and what your reader answers, not the ranking number.

open the explainer → abstract PDF code