The Attribution-Compression Frontier in Retrieval-Augmented Generation
cs.CL · cs.AI
Quick verdict
Measures citation attribution across RAG compressors; finds emitted-grounded gaps (RECOMP 0.86 vs 0.12), rigorously hedged but lacking human calibration.
Summary
The paper measures how context compression in retrieval-augmented generation affects citation attribution, comparing reranking, extractive selection, abstractive summarization (RECOMP), token pruning (LLMLingua-2), and an extract-cluster-rewrite (ECR) construction on ASQA (944 questions) and QASPER (976 questions, 271 papers). Under a fixed generator (Qwen2.5-7B-Instruct) and a DeBERTa-v3 NLI evaluator, it separates emitted citation precision (against compressed text) from grounded precision (against recovered source spans), and contrasts three claim-verification modes: summary-checked, declared-span, and recovered-source. Key results: at nominal budget 0.25 (achieved 0.08), RECOMP's ASQA citations score 0.86 emitted but 0.12 grounded precision (paired gap 0.740, pH=0.0148); recovered-source verification yields a 0.88 unsupported rate versus 0.17 for summary checking; LLMLingua-2's grounded precision degrades from 0.16 to 0.03 across ASQA budgets; only one of eight ECR-versus-extraction precision contrasts survives Holm correction. The paper documents achieved-versus-nominal compression (RECOMP collapses to 0.05-0.08 regardless of budget), a precision-code audit that lowered identity precision from 0.5622 to 0.5086, and a TRUE T5-XXL audit on 200 questions where all 16 emitted-grounded gaps persist. Main limitations, candidly acknowledged: no human calibration, a shared NLI model for span recovery and scoring (circularity), bounded lexical recovery that may understate grounded scores, and a single generator.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- At nominal budget 0.25 (achieved 0.08) on ASQA, a RECOMP-style abstractive compressor's citations score 0.86 emitted precision but only 0.12 grounded precision under the re-attributability protocol (paired gap 0.740, pH=0.0148).
- Recovered-source claim verification for RECOMP yields a 0.88 unsupported rate on ASQA (0.95 on QASPER) versus 0.17 (0.50) when checking compressor summaries, and this gap survives Holm correction.
- Extractive selection maintains grounded precision of roughly 0.42-0.49 on ASQA across budgets while LLMLingua-2 token pruning degrades from 0.16 to 0.03.
- Nominal token budgets misrepresent abstractive compression by up to 32.8-fold (RECOMP collapses to achieved 0.05-0.08 on ASQA), so compression frontiers must be plotted against achieved ratios.
- The emitted-grounded gap persists under a second evaluator (TRUE T5-XXL, 200 questions, all 16 gaps positive at pH=0.0024) but all numbers remain uncalibrated against human judgments.
Strengths
Emitted-vs-grounded decomposition is a sharp operational diagnostic
Table 1 cleanly separates precision against compressor-visible text (RECOMP Pe=0.86 on ASQA) from precision against recovered source spans (Pg=0.12), and Appendix B quantifies the gap for every budget (C001-C008, +0.13 to +0.74), turning a vague worry about summaries into a measurable quantity.
Statistical discipline rare in measurement papers
Every claim is a paired contrast with percentile bootstrap CIs and randomization tests (10,000 replicates, seed 13, paper-level clustering on QASPER), Holm-corrected across all 148 contrasts; the post hoc addition of four Ur-vs-Um tests is documented with a pre-recorded extension plan and family-wide recomputation.
Honest separation of structural artifacts from empirical findings
Achieved-vs-nominal compression correction changes how the field should plot frontiers
Self-audit culture strengthens trust in the numbers
Weaknesses
No human calibration anywhere
Recovery protocol is weak by design and under-sensitivity-tested
Single generator limits all frontier claims
ECR contribution is marginal as evidenced
Answer quality measured only by str-EM on ASQA
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement study of citation attribution under context compression in retrieval-augmented generation, by a single NYU author (arXiv 2609.14245). The pipeline is retrieve - compress - generate - evaluate on ASQA (944 common questions from the ALCE release) and QASPER (976 questions over 271 papers), with a fixed generator (Qwen2.5-7B-Instruct, greedy decoding, ALCE-style numbered citations) and a primary NLI evaluator (DeBERTa-v3-base MNLI+FEVER+ANLI, tau=0.5). Six settings run at nominal budgets {0.5, 0.25, 0.1, 0.05}: identity, cross-encoder reranking, extractive sentence selection, RECOMP abstractive summarization, LLMLingua-2 token pruning, and a proposed extract-cluster-rewrite (ECR) construction. Central claims: (1) emitted citation precision (against compressed text) diverges sharply from grounded precision (against recovered source spans) - RECOMP scores 0.86 vs 0.12 on ASQA at nominal 0.25 (achieved 0.08); (2) three verification modes separate consistency (summary-checked Um=0.17) from declared provenance (Us=1.00, structural) and post-recovery support (Ur=0.88); (3) extractive families hold grounded precision at 0.42-0.49 while LLMLingua-2 falls from 0.16 to 0.03; (4) nominal budgets misstate abstractive compression by up to 32.8x, so frontiers must be plotted against achieved ratios. The paper is unusually careful to frame all numbers as evaluator-dependent estimates.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'attribution laundering' in RAG?
It is the paper's term for when a generated answer's citations are consistent with compressor-generated text (e.g., AI summaries fed to the generator) - earning high citation scores against that compressed context - but the cited text has no locatable link back to the original retrieved sources, so the citation cannot actually be verified against evidence. In the paper's measurements, a RECOMP-style compressor's citations scored 0.86 precision against its own summaries but only 0.12 against recovered source spans on ASQA.
How much does abstractive context compression hurt citation grounding?
Under the paper's NLI-based protocol, quite a lot: a RECOMP-style summarizer on ASQA dropped from 0.86 citation precision against its own summaries to 0.12 against source spans at a nominal 0.25 token budget; recovered-source claim checking found 0.88 of claims unsupported versus 0.17 when checking the summaries. Token pruning (LLMLingua-2) also degraded, from 0.16 to 0.03 grounded precision as budgets tightened. Important caveat: these are evaluator-dependent estimates from one NLI model without human calibration.
Is the paper's proposed method (ECR) an improvement?
Only marginally. The extract-cluster-rewrite construction beats extractive selection in just one of eight Holm-corrected precision comparisons (QASPER, nominal 0.5 budget, +0.063 grounded precision), shows no supported advantage over identity, and its declared-evidence F1 is actually slightly below extraction at every budget (by 0.0007-0.0030). The authors themselves frame it as a setting-specific observation, not a general improvement.
Are the paper's numbers trustworthy?
They are internally rigorous: paired bootstrap confidence intervals, randomization tests with Holm correction over 148 contrasts, a code audit that found and fixed a precision-scoring bug, and a second-evaluator check (TRUE T5-XXL on 200 questions) where all 16 emitted-grounded gaps persisted. However, they are not human-calibrated - the same DeBERTa NLI model both recovers source spans and scores support - so the paper explicitly presents them as evaluator-dependent estimates, not validated rates of source support.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are only available as caption text; curve shapes could not be visually inspected, though all underlying values appear in Appendix tables.
- Code and per-question records are promised 'upon publication' in the Reproducibility Statement; actual availability could not be verified from the text.
- The extraction appears complete (all appendices and both comparison tables present), but any supplementary material outside the arXiv text was not visible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.