Index code memory instead of re-reading whole files

A theme runs through today's picks: cutting what an agent re-derives from scratch down to just what matters. One paper complicates that — model agreement looks like confidence, but often isn't.

Today's pick

-51.5%

fewer tokens burned gathering context before a bug got fixed, with the fix-success rate basically unchanged (one host agent; the other saw -38.9%)

ContextSniper: AntTrail's Token-Efficient Code Memory for Repository-Level Program Repair

When a coding agent fixes a bug in a big repository, it typically reads whole files and scrolls long terminal output, burning its budget on noise before reaching the lines that matter. This paper indexes the codebase and the agent's own action history, ranks evidence, and filters tool output into a compact packet, the full file one lookup away if needed. Tested on two coding agents on real GitHub bug-fix tasks, tokens fell 51.5% and cost 36.4% for one agent (38.9%/27.3% for the other), fix rate unchanged — worth copying if your agent still re-reads whole files for answers a lookup could give.

open the explainer → abstract PDF code

48%

of a frontier model's most-agreed-on answers were still wrong, despite ≥80% sample/judge agreement

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

If your eval pipeline treats "the judges agree" as a proxy for correctness, this large study says that's weaker than it looks. Across 265,000 samples from 53 test runs on a science quiz and a math benchmark, agreement predicted correctness only loosely, and it flips for the best models: the most consistent frontier model agreed with itself on 77% of hard questions, yet 48% of those unanimous answers were wrong. Agreement works better on mid-tier models than as a stand-in for truth — don't wire it into a pass/fail gate unchecked.

abstract PDF

up to -47%

fewer tokens generated by routing on a cheap pre-reasoning confidence check, with no drop in forecast accuracy

What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

When an LLM writes a reasoning chain before forecasting, it's tempting to assume that reasoning produced the answer — this paper shows that's often false. Probes on the model's internal activations predict which way a forecast will move 84% of the time, often before it writes a word of explanation: the answer was largely locked in before reasoning began, so the reasoning can quietly omit evidence that swayed it. Don't trust chain-of-thought as an audit trail unchecked. Routing on that cheap probe let the model skip unneeded reasoning, cutting tokens up to 47% with no accuracy loss.

abstract PDF

+8.3 pts

more tasks solved on a long terminal-session benchmark once a second agent proactively reminds the main one before context decays

Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents

Long-running agents forget things not because information leaves the context window, but because it gets buried under everything since — a failure mode this paper calls "behavioral state decay." The fix doesn't touch the main agent: a second memory agent watches the same trajectory, keeps a structured record of what matters, and injects a reminder only when decision-relevant state has decayed out of reach. That lifted task success 8.3 points on a long terminal-session benchmark and 6.8 points on an airline-booking benchmark, with no retraining. A bolted-on watcher-agent can be cheaper than fixing an agent's own memory.

abstract PDF

up to +39 pts

more functional-and-secure code completions on vulnerability types never seen during tuning, added to model weights with no fine-tuning loop

Functional and Secure Code Generation with Task Vectors

LLMs asked to write code often trade correctness against security, or fix one with a separate pass and risk breaking the other. This paper borrows a trick from AI-safety alignment work: task-vector arithmetic, computing the weight-space direction that made a model more secure and adding it onto its weights, no fine-tuning loop needed. Across six coding models from three families, it raised functional-and-secure completions 2.1 to 36.0 points over the base model, up to 39 on vulnerability types never seen during tuning, decoding speed within 0.6% of baseline.

abstract PDF

+21.6 pts

more accurate at pinpointing the exact step where a coding agent's run went wrong, versus the next-best method (no reference fix available)

TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems

When a coding agent's run fails, the trace is usually hundreds of steps long and full of noise — repeated file dumps, verbose logs — making it hard even for another LLM to spot where things went wrong. This paper folds away the low-signal parts of the trace, then uses hints from the failing test's own error report to point an investigating agent at the likely failure region, while still letting it unfold any section on demand. On 102 real coding failures with human-labeled root causes, that raised localization accuracy 21.6 points over the strongest baseline without a reference fix (10.8 points with one).

abstract PDF