Spec checklists beat bigger test budgets

The standout today is almost embarrassingly simple: paste the spec into the prompt that writes your tests. The rest of the day is about running agents cheaper and catching the ways they fail without telling you.

Today's pick

+38 pts

more correct code (percentage points, every tier tested)

Specification Grounding Drives Test Effectiveness for LLM Code

When an LLM writes code, then tests, then fixes the code until the tests pass, the tests are usually guessing at what the code is supposed to do. This paper shows the fix is one prompt line: paste the task's requirements into the test-writing prompt as a numbered checklist, one test per requirement. That beats every brute-force alternative — 38 points more correct code on every Claude tier (GPT +28, Gemini +19), while doubling the test budget or generating 8 independent test suites barely moves the needle. Tests that could see the spec caught 27 of 30 real bugs; tests written from a careful plan without it caught 2. And they stopped failing correct code: false alarms fell from 33% to zero.

open the explainer → abstract PDF

+11 pts

exact-match from scaling the planner — the executor adds just +2.6

Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents?

In a search agent where one model plans the work and smaller ones fetch and read documents, the planner is what matters: making it bigger buys about 11 accuracy points, making the workers bigger buys 2.6. So put your best model on planning and run the grunt work on cheap models — their fine-tuned 1.7B worker matched a frontier model with 37% fewer tokens.

abstract PDF code

+12.4 pts

task success from deterministic pre-write gates

Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents

A tool-using agent can put your system into a state that breaks business rules while every tool call returns success — no error, nothing to alert on. That was 78% of failures on an airline-booking benchmark. The fix is plain code: validate each proposed write against current state before executing it. Success jumped from 29.6% to 42.0% with no model changes.

abstract PDF

>70%

per-incident agent cost cut in production

Progressive Crystallization: Turning Agent Exploration into Deterministic, Lower-Cost Workflows in Production

Instead of paying an agent to re-solve the same incident every time, this production system spots tasks the agent has solved the same way repeatedly, freezes that solution into an ordinary deterministic workflow, and demotes it if it regresses. After 8 months, 45% of executions ran with no agent at all and per-incident cost fell more than 70%.

abstract PDF

1.4×

agent success from de-noised failure traces

From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization

Feeding raw failure logs to an LLM and asking it to improve your agent overfits to noise. STRACE first groups similar failures and keeps one example of each, then strips the steps that didn't cause the failure, so the optimizer sees only clean evidence. Agent success rose from 42.5% to 58.5%, beating expert-written baselines.

abstract PDF code

From Atomic Actions to Standard Operating Procedures: Iterative Tool Optimization for Self-Evolving LLM Agents

Agents repeat the same multi-step routines across tasks. EvoSOP finds those routines in the logs and packages each one as a single higher-level tool the agent can call, with a merge-test-prune lifecycle that keeps the tool list from bloating. A promising blueprint for MCP toolsets — but the abstract reports no figures and there is no code.

abstract PDF

The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents

Many agent systems use an LLM judge to decide which learned skills to keep. If that judge sometimes passes failures, skill pruning doesn't degrade — past a sharp threshold it silently stops working, and no aggregate metric shows it. The cheap defence: feed the judge outputs you know are bad and measure how often it passes them, before trusting the loop.

abstract PDF