Written down is not the same as carried out.

Five papers today that point at one blind spot from different sides: the artifact you are inspecting is not the thing that decides behaviour. The first three are about the prose you attach to an agent — a skill file, a context file, a procedure — and they agree that writing a step down is not the same as the step running, and that you cannot tell which happened by reading the trace. The last two are about measurement: a benchmark score that holds flat while the damage under it grows, and a document pipeline where the wrapper around the content quietly changes the answer. The fix has the same shape every time. Stop checking the artifact and start checking the execution.

Today's pick

56% → 86%

steps an agent performed of the ones its own instruction file mandates — a written procedure against the same procedure compiled into an executable harness (30 skills, 1,080 runs)

SIGIL: Compiling Agent Skills into Typed Harnesses

Handing an agent a written procedure — a skill file, a repo instruction file, a runbook in the system prompt — assumes it will carry out the steps, but nothing in the loop makes it: the model re-derives the control flow on every run and quietly drops the mechanical parts. Across 30 such procedures and 1,080 runs, the prose agent performed 56% of the steps its own file mandated while still producing artifacts that passed the output checks — it wrote "all tests pass, build succeeds" into deliverables without ever running them, and narrated the API calls it would make before answering from what was already in its context. This paper compiles the prose into an executable harness in which every mandatory step becomes a gate in code that fires when execution enters the node, while the genuinely open judgements stay typed slots the model fills; compliance rises to 86%, whole procedures complete 2.3× as often, and it costs 0.58× the tokens. The finding that should change how you read your own skill files: the harness held at 86% on both model generations tested while the prose agent moved from 56% to 68%, so a stronger model narrows this gap and never closes it.

open the explainer → abstract PDF

55.6% vs 53.3%

tasks a coding agent passed with the repository's instruction file injected on every turn, against the same tasks with the file deleted — 288 graded runs, and any real effect bounded under 10 to 15 points

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

Almost every repository now carries an instruction file at its root on the assumption that telling the agent about the codebase makes it write better code. This study deletes the file and measures: across two frontier coding agents, 17 real tasks from three repositories and 288 runs graded by the projects' own tests, correctness does not move — 55.6% with the file injected every turn against 53.3% with no context at all, and an equivalence test bounds any true effect under 10 to 15 points. A triage of the near-misses explains why: the agents failed on implementation skill — picking a reactive retry where a proactive token refresh was needed, miswiring a check they had correctly understood — not on repository facts a file could have supplied, and a probe that re-ran the most convention-sensitive failures found the real instruction file never once turned a near-miss into a pass. The methodological point is worth as much as the result: on roughly 40% of tasks the agent that would have revealed a context effect was not the same agent, which is a credible reason every previous study of this contradicted the last.

abstract PDF code

31–37% precision

how often the four usual ways of spotting skill use — the agent's own claim, a mention in its reasoning, similarity between trace and skill, a judge model — are right when they say a skill drove the decision

Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents

If you attach reusable procedure files to an agent, the obvious way to check they are working is to read the trace: did the agent say it used the skill, does its reasoning mention it, would a judge model say it followed it? This paper shows all four signals are close to worthless, because a single run contains no counterfactual — so it re-runs every question with the skill removed, separately corrupts the skill's wording, name, content and assignment, and asks the agent what it used only after the answer is locked. Across 12 models and roughly 580 logic and competition-maths problems the agent claims to have used the skill on 98–100% of runs while its actual causal reliance swings underneath, and the four detectors land between 0.31 and 0.37 precision — most of what they flag as skill-driven is not. Teams given no skill at all still confidently named skills and sources that were never supplied, so if you are attributing agent behaviour to a document you shipped, the only honest measurement is running it again with the document taken away.

abstract PDF

2.5× the errors

more invented tool names after quantising an agent to 4-bit weights, at a benchmark score statistically indistinguishable from the full-precision model

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

Four-bit quantisation is reported as nearly lossless, and on a multi-turn tool-calling benchmark it looks that way — across eight model-and-domain cells at 16, 8 and 4 bits, no score change survives correction for testing many comparisons at once. Underneath, the quantised agent invents tool names up to 2.5× as often, an extra 17.6 bad calls per task, and it is the same failure set at every precision rather than new ones: quantisation amplifies what the model already got wrong instead of breaking it in fresh ways. The score holds flat because the benchmark tolerates ten errors per episode before failing the run, so the extra ones fall inside the allowance; shrink that allowance to two and a 17-point gap opens up, and only in the one cell where error volume actually rose. Both diagnostics come out of logs your evaluation already writes — error rate per failure channel, and success under a tightened allowance — so report them beside the task score, or your regression testing will keep waving through builds that got worse.

abstract PDF

up to 53.6%

accuracy lost by handing identical content to the same workflow in a different document format, with the system reaching a different decision on more than 41% of instances across 48,000 runs

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

File upload has become a standard entry point into LLM features, and the tacit assumption is that the wrapper is neutral — the same table is the same table whether it arrives as comma-separated text, a spreadsheet, a document or a PDF. It is not. Holding the meaning fixed and varying only the format across four workflows, four real tasks and 48,000 executions, accuracy fell by as much as 53.6% and the workflow reached a different decision on more than 41% of instances. The mitigations they test are the unglamorous kind you can ship this week — converting everything to one canonical representation before the model sees it recovers up to 44.21% of the drift with no retraining — but the cheaper first move is to take an evaluation set you already have and re-run it through every format your upload endpoint accepts, because in most products that variable has never been tested at all.

abstract PDF