Your skill file is a suggestion

Skills — prose procedure files loaded into an agent's context — became the standard way to extend agents over the last year, and the labs have spent the months since writing about harness design, context engineering and how to keep a long-running agent stable. Today's papers all attack the same seam in that story: the gap between the procedure you wrote down and the one that actually ran. One measures how much of your skill the agent skips, one shows you cannot tell from the trace whether it used the skill at all, and one finds that the repository context file everybody maintains does not move correctness. The last two move the same problem to your inputs: which retrieval strategy wins depends on corpus size, and which document format you accept changes the answer.

Today's pick

56% → 86%

share of a skill's own mandatory steps the agent actually performs — the same procedure as prose, then compiled into a harness

SIGIL: Compiling Agent Skills into Typed Harnesses

A skill file is prose: a procedure the model reads into context and re-derives from scratch on every run, which means it can quietly skip the verification steps it was told to take while still handing back an artifact that passes your output checks. Across 30 skills, an agent following the prose performed only 56% of the steps its own skill mandated and completed the whole procedure just 28% of the time. Compiling that same prose into an executable harness — where a mandatory step becomes code that fires on entry and a forbidden step becomes a path that simply does not exist — lifted step adherence to 86%, full-procedure completion to 65%, and cost 0.58× the tokens, because the prose runs cheap partly by skipping work. The result worth acting on is the flat line: the harness held at 86% across two model generations while the prose swung from 56% to 68%, which means the guarantee stopped living in the model's capability and started living in structure you own.

open the explainer → abstract PDF

0.31 to 0.37 precision

how often a run flagged as having used a skill actually depended on it, against a 0.34 base rate — every way of reading the trace lands on chance

Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents

Evaluators check whether an agent used a skill by reading its reasoning or asking it afterwards, and this paper shows both signals are empty. It pairs every skill-conditioned answer with a matched no-skill counterfactual, so reliance means something testable — did deleting the skill change the decision — and then intervenes on the skill's meaning, wording, name and body separately to see what the behaviour actually follows. Across 3,600 runs on twelve models, four observational detectors (the agent's own attribution, mentions of the skill in the text, similarity between trace and skill, and an LLM judge) scored 0.31 to 0.37 precision against a 0.34 base rate, meaning a flagged run is no more likely to have depended on the skill than any run picked at random. In multi-agent teams it degrades further: influence propagates after the source is lost, and teams handed no skill at all still named skills and sources that were never supplied, in essentially every run. If you need to know whether a skill is earning its place in the context window, delete it and re-run the same inputs.

abstract PDF

+2.3 pts at best

everything a repository context file bought on one agent across 288 evaluated runs — a gap equivalence testing cannot separate from zero

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

Persistent context files are standard practice for coding agents and the published evidence for them contradicts itself. This is a controlled ablation across two frontier agents, three real Python repositories, 17 tasks and 288 gold-test-evaluated runs, comparing no context file at all, the full file injected on every turn, and a topic-organised wiki the agent is told to consult. None of it moved correctness: one agent scored 53.3%, 55.6% and 55.6% across the three, the other 58.8%, 56.9% and 52.9%, with equivalence testing bounding every pairwise difference under 10 points on one and 15 on the other. The triage explains why, and it is the part to internalise — the agents failed on implementation skill, feature design, pattern selection and exact wiring, not on repository knowledge a file could have supplied, and re-running the two closest near-misses with the real file never once converted a failure into a pass.

abstract PDF code

~10 million tokens

the corpus size where plain keyword search overtakes an agentic file-search loop and then leads at every larger size, by a margin approaching 20 points

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Retrieval approaches are normally compared at a single corpus size, which hides the fact that the winner changes as the corpus grows. This study holds the questions, the reader model and the judging protocol fixed while varying corpus size across 28 nested tiers spanning roughly 450-fold, and finds a crossover rather than a champion: an agent that explores the file system sequentially leads on the smallest corpora but spends 39 times more query tokens, and somewhere around 10 million corpus tokens plain keyword search overtakes it and leads at every larger tier, approaching a 20-point margin at full scale. Dense vector retrieval stays cheap but less accurate throughout, and graph-based indexing hits construction costs that stop it before it reaches deployment scale. Choose your retrieval strategy against the corpus you will actually have rather than the one in the demo, and treat agentic search as something that runs after ranked discovery rather than in place of it.

abstract PDF

up to 53.63% lost

accuracy dropped when identical content arrived as a different document format, with the workflow's decision changing in over 41% of cases

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

Any product feature with a file-upload box treats the document format as a neutral wrapper around the content, and it is not. Across 48,000 workflow executions covering four end-to-end document workflows, four real tasks and four formats, converting the same semantic content from one format to another cost up to 53.63% accuracy and flipped the workflow's decision in more than 41% of instances. The authors also tested cheap mitigations available to whoever is calling the workflow, with no retraining involved, and recovered up to 44.21% of that format-induced drift. If your pipeline ingests uploaded documents, put the same content in every format you accept into your eval set today, because until you do, format is an untracked variable sitting underneath every quality number you have.

abstract PDF