Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems
Maximilian Schall, Sedigheh Eslami, Markus Krimmel, Antoine Chaffin, Louis Milliken, Bo Wang, Denis Bykov
cs.IR · cs.CL
Quick verdict
Perplexity's private 190M-doc, 70k-query agentic retrieval benchmark is impressively designed, but closed data and labels block independent verification.
Summary
The paper introduces Q2D-Web, a benchmark for evaluating first-stage retrievers in agentic RAG at production scale. Built from nine months of Perplexity production traffic, it pairs a ~190M-document web corpus (union of top-5,000 results per query, MinHash-LSH deduplicated) with 69,721 agent-reformulated queries in ten languages (17.7% primary, 82.3% support). Three relevance-judgment sets are provided: agent citations, top-50 production rankings, and a combined set adding LLM judgments (DeepSeek-V4-Flash) over pooled unjudged documents, averaging 99.6 positives per query. Benchmarking 13 retrievers under Recall@1000, pplx-embed-v1-4b leads recall (69.11 on Combined, full corpus) while Nemotron-3-Embed-8B leads Recall@100 and nDCG@10; rankings are stable across judgment sets but vary by domain, language, and query type, with neural retrievers weaker on support queries. BM25 ranks last yet uniquely recovers 14,621 positives, nearly six times the next contributor. An RRF-sampled 31.7% subcorpus preserves full-corpus rankings (Kendall τb=1.00) while inflating recall 3-7 points. The authors keep everything private, offering only a leaderboard, citing contamination concerns. Main limitations: closed data prevents independent audit of labels and scores; the authors' own embedding models are evaluated and win; LLM-judge labels and a 512-token input truncation introduce acknowledged biases; no significance testing accompanies close ranking differences.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Q2D-Web is the first benchmark to jointly provide a ~190M-document corpus, ~70k production-sampled agent-reformulated queries in ten languages, and deep (~99.6 positives/query) reusable relevance judgments (Table 1).
- Retriever relative ordering is largely insensitive to the choice among the three judgment sets, but diverges substantially across topical domains, query languages, and query types.
- An RRF-selected subcorpus retaining 31.7% of the corpus preserves the full-corpus system ranking (Kendall τb=1.00) while inflating absolute Recall@1000 by 3-7 points.
- BM25 has the lowest Recall@1000 (44.77 on Combined) yet contributes 14,621 unique positives, nearly six times the next-highest contributor, arguing for family-diverse ensembles.
- Neural retrievers perform worse on agent support queries than primary queries, while BM25 shows a slight reverse gap (+1.9 points on support).
Strengths
Unprecedented scale combination with a clear gap analysis
Table 1 positions Q2D-Web against 10 collections: 190M documents × 69,721 queries × 99.6 judgments/query vs MS MARCO Web Search's 9,374 queries with one click-label each and BrowseComp-Plus's 830 queries on a 100k corpus. The claim of being first to joint large corpus + large production query set + deep judgments is well supported by the table.
Multi-source judgment design with explicit per-source bias inventory
Section 3.3 names each label source's bias (Citation stops once a claim is supported; Web Ranking inherits its first-stage/reranker stack and a corpus-snapshot mismatch; the LLM judge favors fluent long text) and evaluates all three sets separately, then shows ranking stability (Sec. 4.3).
Clean temporal holdout for subsampling validity
Analyses that matter for practitioners
Honest self-assessment
Weaknesses
Fully closed benchmark makes every number unfalsifiable
Conflict of interest around the winning models
LLM-judge circularity in the error analysis
No statistical inference for close rankings
512-token truncation and support-query dominance shape all results
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Q2D-Web (arXiv:2609.08887; Perplexity AI) is a benchmark paper for first-stage retrieval evaluation in production agentic RAG. The authors claim three firsts: a ~190M-document web corpus paired with 69,721 agent-reformulated queries in ten languages, sampled from nine months of production traffic (Table 1 contrast: MS MARCO Web Search has 100.9M documents but only 9,374 queries with one click-label each; BrowseComp-Plus has 830 queries on a 100k corpus); three reusable judgment sets averaging 99.6 positives per query; and the first evaluation-preserving subsampling demonstrated at this scale. A closed leaderboard replaces data release, justified by training-contamination concerns (Sec. 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is the Q2D-Web dataset publicly available?
No. The ~190M-document corpus, the 69,721 queries, and all relevance judgments are kept private to avoid training-data contamination. Only a public Hugging Face leaderboard is offered, where open-weight retrievers are evaluated by the Perplexity team.
Which retriever performs best on Q2D-Web?
It depends on the metric. On the primary metric Recall@1000, pplx-embed-v1-4b leads (69.11 on the Combined label set, full 190M corpus), while Nemotron-3-Embed-8B is strongest on Recall@100 and nDCG@10 under Combined judgments.
How large is Q2D-Web and what makes it different from MS MARCO?
It pairs about 190 million web documents with 69,721 agent-reformulated queries in ten languages and averages 99.6 positive relevance labels per query. MS MARCO Web Search, the closest prior, has ~100M documents but only 9,374 test queries with a single click-derived label each, and its queries are human-issued rather than agent-reformulated.
Why does the paper argue BM25 is still useful if it ranks last?
Although BM25 has the lowest aggregate recall (44.77 on Combined), it uniquely recovers 14,621 relevant documents that none of the twelve neural retrievers finds — nearly six times the next-best unique contribution — suggesting lexical-neural ensembles could raise recall.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text ends at Appendix A, Table 4 (the pooling-retriever list); any content after that point, such as further appendix sections, was not available in the extraction.
- Figures are extracted as raw number/label streams (e.g., the Figure 7 heatmap grid); figure-level readings in this review rely on the surrounding prose, which is intact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.