Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines
Aviral Joshi, Hanoz Bhathena, Max Nelson, Saket Sharma
cs.CL
Quick verdict
Claims reference-free LLM probing recovers gold that flat RAG retrieval misses; unusually rigorous and honest, though TREC-COVID recall regresses.
Summary
The paper proposes Re:CAP, a reference-free audit loop for retrieval coverage in production RAG. Instead of enumerating relevant documents (infeasible without labels), it probes a deployed pipeline for evidence of misses: an LLM extracts topics from the initial answer and retrieved set, generates entity-anchored gap questions for uncovered facets, retrieves candidates with the same retriever, and uses an LLM judge to label candidates novel, redundant, or off-topic until the topic registry stops growing. On MuSiQue, HotPotQA, MultiHop-RAG, TREC-COVID plus MS MARCO (97-100 queries each), Re:CAP at ~238-378 docs/query beats budget-matched flat BM25 by +6.1 to +29.1 pp and flat hybrid top-500 by +12.9 pp on MuSiQue; 9-29% of recovered gold is unreachable by flat BM25 top-500 (48% on TREC-COVID, 21.2% even by a three-retriever ensemble). Blinded human judges rate 78.9% of ensemble-unreachable documents as adding new information (κ=0.79, n=123) and 73.9% on production traffic. Recall is reproducible to within ~1% (CV 0.47%). Limitations: on TREC-COVID recall is below flat BM25 top-500 (0.221 vs 0.241); the label-free monitoring signals are noisy or saturated; cost is $0.59-$2.05/query with judging at 97% of calls; results depend on OpenAI models, with cross-model judge agreement as low as κ=0.41; no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Re:CAP is a reference-free, six-step audit loop that surfaces retrieval gaps on top of an unchanged deployed RAG pipeline via topic extraction, entity-anchored gap-question generation, and LLM novelty judging.
- On four public benchmarks, 9-29% of Re:CAP-recovered gold is unreachable by flat BM25 top-500 (48% on TREC-COVID), and on MuSiQue it beats flat hybrid top-500 by +12.9 pp recall at less than half the document budget.
- 21.2% of TREC-COVID recovered gold is absent even from an ensemble of BM25, dense, and hybrid top-500 (1,500 docs), and blinded humans judge 78.9% of such ensemble-unreachable documents to add new information (Fleiss κ=0.79, n=123; 73.9% on production traffic, n=180).
- End-to-end recall is reproducible to within ~1% across three independent runs (CV 0.47%), and the entity ledger is the dominant generator mechanism (+8.58 pp LOO on MuSiQue), with the other four guards suppressing worst-case failures rather than lifting average recall.
Strengths
Budget-matched evaluation design that pre-empts the obvious objection
Table 2 compares Re:CAP (238-378 docs/query) against flat BM25 at matched Nq, not just top-500, and §3.3 explicitly disclaims 'better retriever' claims; the unreachable-gold share (29.4%/9.2%/48.0%) is offered as the retriever-independent complementarity signal.
Unusual candor in self-analysis
App. O attributes 98.6% of 20,146 missing-gold documents to 'never surfaced as candidate' vs only 1.4% judge-rejected; Table 20 shows 3 of 5 generator mechanisms contribute near-zero or negative average recall (the entity ledger alone gives +8.58 pp LOO on MuSiQue); the TREC-COVID recall regression (0.221 vs 0.241, Table 13) and Azure content-filter recall depression are disclosed rather than hidden.
Blinded human validation with agreement statistics on the strictest stratum
Reproducibility and cost accounting treated as first-class results
Real production grounding without a special code path
Weaknesses
Headline recall regression on TREC-COVID, reframed via an author-defined metric
The deployable label-free monitoring signals are weak by the authors' own analysis
Heavy judge dependence with material model-version sensitivity
Small and partially unverifiable evidence base
No code or artifact release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Re:CAP (Machine Learning Center of Excellence, JPMorgan Chase) targets the "retrieval coverage problem": production RAG corpora lack exhaustive relevance labels, so answer-side frameworks (RAGAS, ARES) cannot register what the retriever never surfaced. The paper reframes evaluation from enumeration to probing. Its six-step loop (§2.2, Algorithm 1) extracts topics from the initial answer and retrieved set D0, reconciles D0 against the registry, generates entity-anchored gap questions for uncovered facets, retrieves with the host retriever, and applies an LLM novelty judge until the topic registry converges — yielding a gap inventory plus gap-count/gap-rate signals. Central claims: (1) 9-29% of Re:CAP-recovered gold is unreachable by flat BM25 top-500 (48% on TREC-COVID, Table 2/17); (2) +12.9 pp over flat hybrid top-500 on MuSiQue at less than half the document budget (0.901 vs 0.772); (3) 21.2% of TREC-COVID recovered gold escapes even a three-retriever top-500 ensemble, and 78.9% of such documents add information the baseline answer lacks per blinded humans; (4) run-to-run recall CV 0.47%. §3.3 is explicit that Re:CAP is an audit layer, not a competing retriever.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Re:CAP?
A reference-free auditing method for RAG pipelines. It takes a query plus the pipeline's initial answer and retrieved documents, then iteratively generates probing questions about information facets that appear missing, retrieves candidates with the same retriever, and uses an LLM judge to keep only documents adding genuinely new information. The output is a per-query inventory of retrieval gaps plus gap-count and gap-rate monitoring signals — it requires no relevance labels.
Does Re:CAP improve retrieval or just measure it?
It is an audit layer, not a retriever upgrade. In the paper's budget-matched comparisons it reaches higher recall than flat baselines on MuSiQue and HotPotQA at fewer documents seen, but on TREC-COVID its recall is slightly below flat BM25 top-500 (0.221 vs 0.241); there the authors position its value as surfacing gold documents that no flat top-500 run — or even a three-retriever ensemble — reaches.
How much does Re:CAP cost per query?
$0.59-$2.05 per query at default (GPT-4.1) settings depending on corpus, with ~97% of the roughly 250-500 LLM calls being judge calls. Swapping all components to GPT-4.1-mini cuts cost 4.6x for about -0.5 pp recall. Production medians were $1.68-$1.79 per query, with a long tail (max $24.86).
What datasets were used and what were the headline numbers?
MuSiQue, HotPotQA, MultiHop-RAG, and TREC-COVID, plus MS MARCO as a BM25-only sensitivity ladder, with 97-100 queries each. Re:CAP beat budget-matched flat BM25 by +6.1 to +29.1 pp recall; 9-29% of its recovered gold was unreachable by flat BM25 top-500 (48% on TREC-COVID, 21.2% even by a BM25+dense+hybrid ensemble); blinded human annotators judged 78.9% of ensemble-unreachable documents to contain new information (Fleiss κ=0.79, n=123).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Parts of §4.1-4.2 and Appendices A, F, and N.1 are truncated in the extraction; some secondary numbers (e.g., full production worked examples) were unavailable.
- Figures 1-5 were assessed via captions and surrounding text only; visual content could not be verified.
- Model names/versions (GPT-5.2, gpt-5.5, etc.) are as stated in the text and could not be independently verified.
- No code-availability statement was found in the extracted text; if the published PDF carries one in a truncated region, codeAvailable may be understated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.