The standard reliability move in retrieval is to score each passage against the question and throw away the ones that do not support the answer. For any question needing more than one hop that filter is broken by construction: such questions are built so no single passage is sufficient, and the passage carrying the answer is the one the question never names. Separating real evidence from distractors scores 0.643, 0.523 and 0.560 on three multi-hop sets against 0.951 on single-hop — and filtering came out worse than not filtering at all in every configuration tested, by as much as 13.4 points. The repair is to stop scoring against the question you were asked.
A two-hop question names one thing and asks about another, reachable only through it. Who directed the film that won an award in a given year? The first passage tells you which film. The second — the one holding the answer — is about the film, and never mentions the award, the year, or anything else you asked. Score it on its own against the question you typed and it looks like a distractor, because on its own that is exactly what it is. Set the filter's threshold below and watch which passages survive.
There is no threshold that fixes this, and the paper checks. At a threshold permissive enough to be nearly useless as a filter — 0.10 — the scorer has already rejected 84% of the passages that actually carry the answer. The measure the paper reports is threshold-free for exactly this reason: a verifier that cannot rank real evidence above a distractor cannot be rescued by tuning the cutoff.
The number to watch is how well the scorer ranks genuine evidence above distractors, where 1.0 is perfect and 0.5 is a coin flip. On single-hop questions the same pipeline, same scorer, same everything, sits at 0.951. Move to multi-hop and it collapses to somewhere between 0.52 and 0.64. That gap is the whole paper, and the obvious objections — the scorer is too small, the passages are too long, the threshold is wrong, the retriever is bad — are each ruled out in turn.
The sharpest of the seven is the sufficiency test. A genuine pair of passages — everything you need to answer — scores 0.540. Pad that pair with an unrelated passage, making it longer and noisier but still fully sufficient, and the score falls to 0.127. The scorer is not measuring whether the answer is derivable. It is measuring surface overlap with the words in the question.
A bad verifier is not automatically a problem — you could argue it just fails to help. It is worse than that. Run the whole pipeline end to end, comparing a per-passage gate against passing everything through unfiltered, and the gate loses in every dataset, every generator size and both prompt formats tested. Switch datasets below to see the size of it. Then comes the part that should change your roadmap: the penalty grows with generator capability — step the model size up in the second panel and watch both gaps widen together.
The reason is not mysterious. A bigger model is better at ignoring an irrelevant passage and better at chaining two relevant ones — so the cost of handing it noise falls, while the cost of deleting its evidence stays total. Every capability upgrade you ship makes your filter a worse trade. Deleting evidence is not recoverable; tolerating noise increasingly is.
If the failure is that no passage is sufficient for the whole question, the fix is to stop asking each passage the whole question. Break the query into the sub-questions it decomposes into, and verify the later-hop passage against its own sub-question — one that mentions the entity that passage is actually about. Same scorer, same passages, same threshold. Only the hypothesis changes. Switch the source below.
Two details keep this honest. First, the effect is from restructuring the question, not from adding context: naively appending the top passage's text to the query buys a mean of +0.034, and the full-paragraph version actively hurts. Second, an off-the-shelf decomposer with no fine-tuning captures only 31% of the ceiling, and only when it is given a retrieved passage to anchor on — decomposing from the question alone scores 0.533, which is below just using the original question. The decomposition has to be grounded in something retrieved.
Most production retrieval stacks have one of these in them, usually added after a hallucination incident and never re-measured. The question is what share of your traffic actually needs two hops — an entity resolved through another entity, a policy found via the product it applies to, a person found via their team. Set that share and the paper's measured penalties do the rest.
The cheapest experiment, and the one to run before any of this: take your gate's decisions on a day of real traffic, pull the passages it dropped, and re-answer those queries with the drops restored. If the unfiltered answers win, your filter is a hallucination control that is causing the failure it was installed to prevent.