Your agent runs whatever the skill says

Three papers today about a component that made its decision from a stand-in rather than from the thing itself. A coding agent reads a skill's friendly one-line description and runs the script underneath it without ever looking at the script. A model judging two answers is handed a tidy record of the evidence instead of the answers, and gets measurably worse at agreeing with people. A router picks which agent handles a request from surface keywords rather than from what the request actually says. Every one of those substitutions was somebody's reasonable simplification, and every one is where the system now fails.

Today's pick

96% of runs

how often one enterprise coding agent committed to executing a shell command hidden inside a skill file it had been told to load

Towards a Risk Assessment of Malicious Skill Files in Coding Agents

Agent skills are folders of instructions plus optional scripts that a coding agent loads to specialise itself, pulled from marketplaces and shared repositories with roughly the trust you would give a README and none of the review you would give a dependency. This paper turned 471 real attack commands into 2,826 skills whose visible description reads as ordinary project setup — a credential harvester is captioned "Initializing project metadata" — with the payload framed as a mandatory preflight step the agent must finish before it is allowed to start the actual work. Across 5,629 completed runs against two enterprise coding agents, one committed to running the hidden command in 95.5% to 96.1% of runs and the other in 71.6% to 74.0%, near-identical regardless of which of six models wrote the disguise, while explicit safety refusals appeared in just 1.99% of runs overall. Most non-exploited runs were not saves — the agent simply never mentioned the preflight step at all. Treat an installed skill as executable code from an untrusted source, because that is exactly what it is.

open the explainer → abstract PDF code

−4 to −6 pts

agreement with human preferences, when the judge decides from a saved evidence record instead of the answers themselves

Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation

A popular pattern in judging pipelines is to have the model extract criteria and evidence in one call, then reach a verdict from that record in the next — it looks more auditable, more modular, and harder to bias. This paper tests whether the record actually carries what the verdict needs, and it does not: across 24,000 judgments on three preference datasets with two frontier judges, locking the evidence dropped agreement with human preferences from about 76% to between 70% and 72%, and roughly doubled the rate at which the judge flips its answer when you swap which candidate is shown first, from around 9% to between 16% and 19%. Scoring each answer in its own isolated call and then comparing the two frozen write-ups was worse again. The reason is that once the record is the only input, every omission and every framing choice inside it becomes part of the evaluator — so keep persisting evidence for logging and review, but let the deciding call see both the record and the original answers.

open the explainer → abstract PDF

0% against 100%

adversarial routing cases handled correctly by a keyword-matching router, against one that reads what the request says

OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality

Multi-agent pipelines are normally scored on whether the final answer came out right, which tells you nothing about which hop broke or how far the damage travelled. This work injects known faults into templated workflows and measures both, and two findings are worth acting on immediately. First, a router that dispatches on keywords and flags handles every well-behaved case and none of the adversarial ones, where the surface markers are missing or misleading — while a crude baseline that merely reads the task description gets all of them, so the blind spot belongs to the mechanism rather than to the difficulty of the task. Second, retrying only repairs the failures you were least worried about: a failed tool call recovers fully, an ambiguous hand-off recovers 30% of the time, and corrupted upstream state, contradictory sub-agent outputs and out-of-order execution never recover at all — those three corrupt every stage downstream, with the blast radius growing by roughly one stage for each stage of pipeline depth.

open the explainer → abstract PDF