ORDER: Task-Conditioned Routing for Retrieval-Augmented Generation
Aurélien Pellet, Julien Perez, Marie Puren
cs.AI
Quick verdict
Query-conditioned routing (QRe+UMS) lifts multi-hop RAG recall on 1887 French archives (R@3 37.7→53.7); convincing on this corpus, transfer untested.
Summary
The paper argues that a fixed, one-size-fits-all RAG configuration fails on heterogeneous expert corpora and proposes ORDER, splitting optimisation into an offline, document-conditioned axis and an online, query-conditioned axis. Online, a logistic-regression router (QRe) predicts from the query embedding whether the large parliamentary collection is needed (98.9% binary accuracy on the held-out split), while a Uniform Multi-source Sampler (UMS) splits the top-k budget evenly across active collections so one dominant corpus cannot monopolise results. Offline, training questions are clustered with UMAP+HDBSCAN and each cluster receives its best chunking strategy and metadata filter/reranker, assigned at inference by nearest-centroid routing. On HistoriQA-ThirdRepublic (875 historian-validated multi-hop questions over three 1887 French sources), QRe+UMS lifts Recall@3 from 37.7 (naive dense baseline) to 53.7, beating BM25, HippoRAGv2, LinearRAG and Adaptive Chunking, and raises LLM-judged end-to-end accuracy from 31.0 to 48.1; gains persist when composed with BM25 or Adaptive Chunking. The authors candidly show that task-conditioned chunking gains are absorbed once routing is active. Main limitations: a single corpus and year, a single embedding family, LLM-generated benchmark questions, LLM-as-judge evaluation without human adjudication, and an unquantified precision cost paid by UMS for its coverage gains.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Query-conditioned routing (QRe) plus uniform budget allocation (UMS) raises Recall@3 from 37.7 to 53.7 over a naive dense baseline on HistoriQA-ThirdRepublic, beating BM25, HippoRAGv2, LinearRAG and Adaptive Chunking (Table 4).
- The mechanism is retriever-agnostic and composable: BM25 + QRe&UMS lifts Recall@5 from 30.7 to 57.0, and routing is additive on Adaptive Chunking (37.2 → 59.7 Recall@10).
- Task-conditioned indexing helps the naive retriever (+5.4 Cov@3 chunking, +8.9 metadata), but the chunking gain is fully absorbed by routing; only the metadata gain persists under UMS (+6.0, p < 0.05, Table 6).
- Query-conditioned RAG needs zero LLM calls at indexing and query time, dominating LLM-indexed graph RAG on both quality and offline cost (Table 7).
- End-to-end LLM-judged answer accuracy rises from 31.0 to 48.1 with QRe & UMS (Table 5).
Strengths
Unusually candid ablation and negative-result reporting
§5.2 + Appendix B.4 show TC-Chunking's +5.4 Cov@3 under the Naive Retriever is a pure noise-suppression effect, fully absorbed once QRe is active (∆ = −0.002 ± 0.002 on the mechanically-active subset, robust to score-k sweeps 3→9); Appendix I reports the 14×14 joint grid never beats single-axis TC-Metadata, with a credible winner's-curse explanation.
Informative router error analysis
The 3-class → binary collapse is justified quantitatively: 82 of 87 3-class errors land between I+D and G+D, which share the same active collection set (Tables 3, 18); only 2/287 Débats questions are misrouted destructively; binary accuracy (285+148)/438 = 98.9% (Table 16), per-class F1 given (Table 17).
Controlled comparison discipline
Strong cost-quality positioning
Composition results are the real evidence of generality
Weaknesses
External validity rests on one corpus, one year, one embedding family
Benchmark circularity risk beyond what the authors admit
End-to-end evaluation is single-model, self-judged, untested
Internal inconsistency in the split protocol
Per-cluster conclusions on thin support
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ORDER targets a real, well-motivated gap: RAG over heterogeneous expert corpora where a single indexing/retrieval configuration cannot serve all query types. The setting is concrete and unusual — three 1887 French Gallica collections (the parliamentary transcripts *Les Débats*, ~30× larger than the two newspapers combined, plus *Le Gaulois* and *L'Intransigeant*) with 875 historian-validated multi-hop questions from HistoriQA-ThirdRepublic. The paper makes three claims: (1) a supervised router (QRe) plus a uniform multi-source sampler (UMS) attack distinct failure modes (source contamination vs starvation) and jointly lift Recall@3 from 37.7 to 53.7 over a naive dense baseline, beating BM25, HippoRAGv2, LinearRAG and Adaptive Chunking (Table 4); (2) query-conditioned indexing — per-cluster chunking and metadata configurations — improves the naive retriever; (3) this comes at near-zero online cost, with no LLM calls at indexing or query time (Table 7). An end-to-end result (31.0 → 48.1 answer accuracy, Table 5) supports claim (1) downstream.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ORDER in RAG?
ORDER (Optimal Routing for Dynamic Evidence Retrieval) is a query-conditioned retrieval-augmented generation framework. Instead of one fixed index and retrieval setup for all questions, it routes each query to the index and source collections that suit it: a small logistic-regression classifier (QRe) decides which collections to query, a uniform sampler (UMS) splits the retrieval budget evenly among them, and per-cluster chunking/metadata configurations chosen offline are assigned to new queries by nearest-centroid routing.
How much does ORDER improve retrieval over a standard RAG pipeline?
On the HistoriQA-ThirdRepublic benchmark (875 multi-hop questions over three 1887 French sources), combining QRe and UMS raises Recall@3 from 37.7 with naive dense retrieval to 53.7, and LLM-judged end-to-end answer accuracy from 31.0% to 48.1%. It outperforms BM25, HippoRAGv2, LinearRAG and Adaptive Chunking on the same held-out split, and adding the routing layer lifts BM25's Recall@5 from 30.7 to 57.0.
Does ORDER need an LLM at indexing or query time?
No. Unlike graph-based RAG systems that run an LLM over the corpus to build a graph, ORDER's adaptivity is paid in embeddings, clustering (UMAP+HDBSCAN) and a linear classifier; the only generative call is the final answer generation, same as naive RAG. An LLM (Claude Opus) was used once, offline, only to help draft the regex chunking rules and metadata schema, validated by a historian.
What are the main limitations of the ORDER paper?
All results come from a single corpus and year (1887 French sources) and a single embedding family (Cohere embed-v4.0); benchmark questions were LLM-generated and only historian-filtered; end-to-end scoring used the same model as generator and LLM judge with no human adjudication; the precision cost of the uniform multi-source sampler is not quantified; and transfer to other domains is argued, not demonstrated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several sections were elided in the provided extraction (parts of §4.4.2/§4.5, mid-references, parts of Appendix C.1, end of Appendix G); all main quantitative tables (3–7, 9–11, 14–21) were intact.
- Figure 1 (UMAP Voronoi visualisation) is available only as caption text; its content could not be inspected.
- Cluster IDs in Appendix C differ from those in Table 10 (likely different seed runs); the exact seed alignment between the qualitative analysis and the averaged tables is unclear from the text.
- Claims about baseline configurations rest on the authors' Appendix H description; the released repository was not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.