Two agents, one blind spot

Three papers today about the same assumption, made in three places and wrong in different directions. A pipeline is certified by multiplying each agent's reliability, which assumes the agents make separate mistakes — they do not, when they share a model. A prompt is loaded with rules one at a time, each of which the model handles, which assumes they will hold together — they do not, because each one really is its own coin toss. And a coding agent's command is scored where it was written rather than where it runs, assuming the interface between is neutral plumbing.

Today's pick

90.0%

of the missions where either agent failed, both failed — two instances of one model in a handoff

Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence

The usual way to state how reliable a multi-agent pipeline is: bound each agent, multiply, report the product. That step assumes the agents fail for unrelated reasons, and a reviewer checking a writer is very often the same model with a different prompt. Across 18,000 missions scored by deterministic code with no model in the judging loop, two instances of one model failed together on 36.3% of missions where multiplying predicts 14.6%, and swapping the second agent to a different model cut the association in six of six comparisons — while changing vendor, model already different, did nothing. The error runs one way and it runs against you: chains are fine, but every redundant design, a checker on a writer or a 2-of-3 vote, gets credited with independent evidence it is not supplying. If one model sits on both ends of a double-check, measure how often they fail on the same input before counting the second one.

open the explainer → abstract PDF code

5.7%

of responses satisfied all eight rules at once, from models that pass each one alone about 41% of the time

Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction

Every rule you add to a system prompt is cheap on its own; the joint bill arrives later. Across 15 models and 369,753 checks, each rule graded by code rather than by another model, per-rule compliance decayed gently — each added rule multiplies the average pass rate by about 0.92 — while the chance of getting all of them right at once collapsed: at eight simultaneous rules models satisfied each one about 41% of the time and all eight 5.7% of the time. There is no clever fix because the failures are almost uncorrelated (mean correlation +0.067 across rule pairs), so nothing is interfering with anything and re-pairing or reordering buys nothing; only per-rule reliability moves the ceiling, which sits at five or six rules for the best models and at three or fewer for 12 of the 15. Count the rules that must all hold, check them with code rather than a partial-credit score, and split the work into separate calls when the count is high.

open the explainer → abstract PDF

55 to 73 pts

of success lost when the identical command is replayed through one extra layer that re-reads its quotes

QuoteBench: How Matched Scores Can Hide Command-Path Failures

A coding agent writes a shell command, and something between the model and the shell usually re-reads it: a remote wrapper, a container exec, a CI run step. Replaying the same stored command through one added double-quoted parser cost every one of eight model configurations 55.4 to 73.2 points of success on 56 exactly-validated tasks, and not only on nasty payloads — the ordinary control tasks lost 28.6 to 57.1 points, because models emit backticks, dollar signs and quotes in perfectly normal commands. One sentence telling the model where its reply will run recovered 30.4 to 60.7 points for six of the eight, which is how an aggregate score stays flat while hiding two large opposing effects: the strongest configuration's −3.6 point gap was −64.3 of damage against +60.7 of compensation. Escape the command at the interpolation point or hand it over as a temporary script, and benchmark models on the path you actually deploy.

open the explainer → abstract PDF code