The memory returns the value you already changed

Three papers today about the state an agent drags from one turn to the next. The first shows memory systems retrieve what was said rather than what is still true, and that making the model write out a fact's revision history before answering lifts six existing backends by 32 to 67 points. The second asks what a memory system should commit to disk at all, and finds models re-check stale facts but almost never ask the user what they meant. The third is the other half of the window — the documents you load to give an agent a capability — where one plausible near-miss costs 23 points of task success.

Today's pick

+32 to +67 pts

more answers reflecting the current state rather than a superseded one, when a prompt-level wrapper is bolted onto six existing memory and retrieval systems — no extra model calls

Can agent memory systems track evolving state?

Ask an agent about something the user revised three sessions ago and it often hands back the version you superseded — the budget before it was raised, the rule before its exception lapsed — because memory systems retrieve what was said rather than track what is still true. This benchmark runs 234 multi-session scenarios past memory systems, retrieval baselines and plain long context, grading each answer three ways: current state, the specific stale value someone who missed the revision would land on, or neither. The best configuration in the main table is right 36% of the time, and what moves the number is not a new store: a prompt wrapper that makes the model write out each fact's value chain — first value, every revision with its turn number, current operative value — before answering lifts six existing backends by 32 to 67 points, of which 15 to 32 survive a length- and cost-matched control. Add that trace step to your memory read path before shopping for a memory system.

open the explainer → abstract PDF

0 of 12 vs 12 of 18

ambiguous updates an open-weight agent asked the user about, versus facts that could have gone stale it correctly re-checked — same model, same prompt, one bare instruction listing both options

Remember, verify, or ask? Memory commitment in LLM agents

An agent with persistent memory has four moves when a user says something: keep it forever, use it for this task only, re-check it against a source later, or ask which was meant. This benchmark builds 140 scenarios where exactly one move is right, then scores both the action the model names and the tool call it emits. Models re-check changing facts tolerably and essentially never ask — the open-weight model asked on none of the 12 ambiguous items while verifying 12 of 18 freshness ones — and its stated choice matched its own call only 23% of the time, 57% for both Claude models. Five written commitment rules with a tie-break toward the weaker action cut wrong durable writes from 24% to 10%; four worked examples moved accuracy from 0.557 to 0.771 and left the ask rate at 0.333. Grade the tool call, not the reasoning, and expect to build the ask path by hand.

open the explainer → abstract PDF

0.73 vs 0.20–0.52

share of coding tasks solved when the documents loaded into context were chosen as a whole set under a token budget, versus by released skill routers, text retrievers, or letting the model pick for itself — on 28% fewer tokens

Choosing which skill documents to load, as a set

Agents pick up capabilities by loading skill documents into the context window, and almost every router scores each document alone and packs the top few until the budget runs out. On a code benchmark unsolvable without the right documents, the set matters far more than the ranking: two complementary documents reach 93% success, a redundant third buys one point for 225 tokens, and one that is semantically related but irrelevant costs 23 points. Their selector scores whole sets — capability coverage with diminishing returns, minus a flat charge per token — and walks greedy chains by benefit-per-token while keeping every partial set, hitting 0.73 task success against 0.20 to 0.52 for released routers, retrievers and the model's own picks, on 28% fewer tokens. If you inject docs, tool definitions or examples by relevance rank, delete the near-misses and measure.

open the explainer → abstract PDF