An agent with persistent memory has four moves when a user says something: write it down forever, use it for this task only, re-check it against a source later, or ask which of those was meant. This benchmark builds 140 scenarios where exactly one move is right, then grades both the action the model names and the tool call it actually emits. Models re-check changing facts tolerably and essentially never ask — one open-weight model asked on none of the 12 ambiguous items while verifying 12 of 18 freshness ones. Worse, naming an action and emitting a call are not the same decision: they agree 23% of the time on that model and 57% on both Claude models.
Every scenario has the same three parts: an acquire context where the information first appears, a candidate update the system might store, and a later reuse context where that stored thing surfaces and either helps or quietly distorts the answer. The whole benchmark hangs on one distinction — which of four actions the agent should take — so step through them and see what separates each from the one beside it.
Base rates matter for reading everything after this: across the 140 audited scenarios the right move is persist 38 times and use-now 40 times, against verify 33 and ask 29. Eight of the items are lexical traps that put words like "always" or "today" in contexts where the heuristic those words suggest is exactly wrong.
The paper's central cross-family finding is a lopsided pair of recalls. Given a fact that could have gone stale, models mostly go and check. Given an update whose scope, durability or referent is genuinely unresolved — where the user is the only authority — they commit anyway rather than ask. Switch models below with the bare instruction that simply lists the four actions and nothing else.
The gap is not a capability ceiling — every model can produce the ask when told to. It is a default. Checking a source is an action the model takes on its own; asking is an action that interrupts the user, and nothing in a bare instruction makes that trade look worth it.
Three conditions, identical scenarios. Bare defines the four actions and stops. Few-shot adds one worked development example per action, four in total. Policy adds five written commitment rules including a tie-breaker: when persisting and a weaker action are tied, take the weaker commitment. They do not buy the same thing, and the difference is the practical result in the paper.
On the open-weight model the two interventions split cleanly: four examples move accuracy 0.557 → 0.771 and the rules do not move it significantly at all, while the rules cut wrong durable writes 0.243 → 0.100 and the examples leave them at 0.129. If your failure mode is a polluted memory store rather than a wrong answer, the rules are the intervention.
The second evaluation mode replaces "which of these four would you do" with an actual structured call: memory_write, use_now, check_source or ask_user, each carrying a required string argument checked by deterministic minimum-content and relevance rules. Same models, same scenarios, same bare instruction. Accuracy drops on every model, and the disagreement is not a rounding error.
Thirty of seventy decisions change on both Claude models when the same question is asked as a tool call instead of a label — and their raw agreement is 0.571 against 0.229 for the open-weight one. Whatever your evaluation harness grades, if it is not the call your runtime actually executes, it is measuring a different system.
The shape in production: a stream of user statements, some worth keeping forever, some scoped to one task, some that will be stale in a month, some genuinely ambiguous. Set your own volume and mix. The per-condition rates applied are the paper's held-out numbers for the open-weight model, which is the one whose defaults are furthest off.
Note which column never gets close to acceptable. Even the best condition on this model asks about a third of the items it should, and on the combined 140-item set the best clarification recall anywhere is 0.519. Plan for the ask path to be code — a confidence threshold, an unresolved-referent check — rather than an instruction.