Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering
Kevin Mo, Nathan Mo, Richard Zhu
cs.CL · cs.AI · cs.IR
Quick verdict
Claims nearly half of MuSiQue per-hop failures lack facts in retrieved passages; mostly convincing analysis, though key statistical claims need scrutiny.
Summary
The paper argues that failures in multi-hop question answering are routinely misattributed to retrieval: beyond retrieval failures (the gold passage is not retrieved), there is a second mode — extraction failures — where the gold passage is retrieved but does not state the fact the current reasoning step requires, a phenomenon the authors call the fact-grounding gap. Using a Self-Ask pipeline built on GPT-4.1-mini with BM25 and Contriever retrieval over MuSiQue, HotpotQA, and 2WikiMultihopQA, they judge per-hop fact presence with an LLM judge validated against 200 human annotations (Cohen's κ = 0.840). On MuSiQue dev (6,404 hops), only 49.1% of hops are answerable from retrieved passages; extraction failures account for 27.3% of hops — 47% of per-hop deficiencies — and persist under dense retrieval and under every retrieval intervention tested: augmentation and reranking lift accuracy on retrieval-failure hops (+8.6% and +25.3%) but leave extraction-failure hops unchanged (−0.5%), and a corpus audit finds the needed fact absent corpus-wide in 97.7–100% of such failures. A DeBERTa-v3-large predictor (F1 0.785) enables targeted re-retrieval that roughly matches intervening at every hop at half the retrieval cost, with McNemar-significant gains on MuSiQue and HotpotQA but none on 2WikiMultihopQA. Limitations include a single reasoner model, reliance on an LLM judge with only 200-item human validation, an unexplained inconsistency between the per-hop (~9%) and aggregate (27.3%) extraction-failure rates, and the absence of significance tests for the targeted-versus-every-hop comparisons.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On MuSiQue dev, 27.3% of 6,401 hops are extraction failures — the gold passage was retrieved but does not state the needed fact — constituting 47% of all per-hop deficiencies.
- Extraction failures persist under Contriever dense retrieval and are unaffected by augmentation or reranking (−0.5% accuracy on such hops vs +8.6%/+25.3% on retrieval-failure hops), establishing a ceiling for retrieval-only improvements.
- A corpus-wide search finds the needed relational fact absent from every passage in 97.7–100% of extraction failures, indicating genuine corpus gaps rather than retriever weakness.
- A DeBERTa-v3-large fact-presence predictor (F1 0.785 on MuSiQue dev, trained on 46,610 LLM-judge labels) enables targeted re-retrieval matching every-hop intervention at roughly half the retrieval calls, with McNemar-significant gains over baseline (p < 0.001) on MuSiQue and HotpotQA.
- Standard retrieval metrics cannot detect this failure mode: among incorrectly answered hops, retrieval recall and fact presence are nearly uncorrelated (Spearman ρ = 0.042, Appendix D).
Strengths
Novel per-hop diagnostic with a validated measurement instrument
The two-mode decomposition (Table 2: 30.7% retrieval failure, 27.3% extraction failure, 42.0% fact present over 6,401 MuSiQue hops) is invisible to document-level metrics, and the judge is validated against humans (κ = 0.840, 92.0% agreement, §4.1) with the conservative-bias direction disclosed (12/16 disagreements).
Corpus audit elevates the finding beyond judge opinion
§6.5 searches the full corpus for each extraction failure and finds the relational fact absent everywhere in 97.7% (MuSiQue), 100% (HotpotQA), 97.6% (2Wiki) of cases — strong evidence the gap is a corpus property, not retriever weakness.
Retriever robustness check is properly designed
Interventions validate the taxonomy end-to-end
Honest reporting of negative results and cost
Weaknesses
Internal numeric inconsistency in a load-bearing sentence
Key cost-equivalence claim lacks statistical support
LLM judge is the ground truth for the central construct
Severity profile is MuSiQue-centric despite general framing
Underspecified per-hop construction outside MuSiQue
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper diagnoses where multi-hop QA pipelines break, distinguishing **retrieval failures** (gold passage not retrieved) from **extraction failures** (gold passage retrieved but lacking the fact the current hop needs) — the 'fact-grounding gap.' Central quantitative claims: on MuSiQue dev (2,417 questions, 6,404 hops), only 49.1% of hops are answerable from accumulated retrieved passages (Table 1); the three-way outcome split is 30.7% retrieval failure / 27.3% extraction failure / 42.0% fact present (Table 2), so extraction failures constitute 47% of per-hop deficiencies. Fact presence degrades from 68.5% at hop 1 to 18.5% at hop 4. The authors further claim retrieval-only fixes cannot close the gap (§6.5: −0.5% on extraction-failure hops vs +8.6%/+25.3% on retrieval-failure hops under augmentation/reranking), backed by a corpus audit (97.7–100% corpus-wide fact absence) and a Contriever robustness run (§4.3). Abstract framing is mostly scoped ('as much as 47%'), but generality claims outrun the MuSiQue-heavy evidence (see Risks).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the fact-grounding gap in multi-hop QA?
It is a failure mode where a QA system successfully retrieves the correct supporting passage for a reasoning step, but that passage does not actually state the specific fact the step needs — so the step is unanswerable even though document-level retrieval metrics report success. The paper reports it on MuSiQue, HotpotQA, and 2WikiMultihopQA.
How common are extraction failures compared to retrieval failures?
On MuSiQue dev (6,401 hops), 30.7% of hops were retrieval failures (gold passage missing) and 27.3% were extraction failures (gold passage retrieved, fact absent), so extraction failures make up about 47% of all per-hop deficiencies. Severity varies by dataset: only 3.1–4.5% of HotpotQA hops and 2.7% (comparison) to 16.6% (multi-step) of 2Wiki hops show the problem.
Can better retrievers or more retrieval fix extraction failures?
Not in this study. Switching BM25 to the dense retriever Contriever did not reduce extraction failures (598 → 725 on a matched MuSiQue criterion), and re-retrieval interventions (question rephrasing augmentation, cross-encoder reranking) improved accuracy on retrieval-failure hops (+8.6% to +25.3%) but not on extraction-failure hops (−0.5%). A corpus-wide audit found the needed fact absent from all passages in 97.7–100% of these cases.
Is code and data available for this paper?
Yes. The paper links a GitHub repository (RichardZhu123/fact-grounding-gap); all three datasets are public, the reasoner is a pinned OpenAI model (gpt-4.1-mini-2025-04-14), and the authors report modest costs (~$80 GPU training, ~$100 API usage).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 appear only as embedded text in the extraction; the exact per-depth numbers behind Figure 3 cannot be independently verified.
- Appendices A–K were fully extracted and no sections appear missing, but flattened table formatting required careful row-label mapping (e.g., Table 7 condition labels and the Table 13 predictor column).
- arXiv v1 (15 Sep 2026); peer-review status is unknown.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.