TRACE: Accountable Agentic Retrieval for Source Discovery in Digital Archives
Donghan Bian (ENC, LRE), Marie Puren (LRE, Florian Cafiero (LRE, ENC)
cs.AI
Quick verdict
TRACE claims R@10=0.856 over graph/agentic RAG on 1887 French archives; strong ablations, but single benchmark, unmatched budgets, no significance tests.
Summary
TRACE is a training-free agentic retrieval framework for source discovery over digitised historical archives, developed within the DECIDON project on French Third Republic political discourse. The pipeline decomposes a question into corpus-targeted sub-questions, seeds a bounded agent loop with a reciprocal-rank fusion of BM25, dense and temporal channels, and has the agent issue explicit accept/reject/hold verdicts on retrieved chunks; held items are re-evaluated once full evidence accumulates, and a final count-grouped reranker orders documents using the agent's own acceptance rationales. Evaluation uses HistoriQA-ThirdRepublic: 1,752 French questions over 3,386 OCR-degraded parliamentary and newspaper documents from 1887. With a DeepSeek-V4-Flash backbone, TRACE reaches R@10 = 0.856 and MRR = 0.653, exceeding BM25 (0.643), dense retrieval (0.734), graph-based HippoRAG 2 (0.737) and LinearRAG (0.727), and agentic baselines A-RAG (0.385) and MA-RAG (0.586), with the largest margins on comparative multi-hop questions (0.674 versus 0.302 for dense retrieval). A cumulative ablation attributes the biggest share to question decomposition (+8.9 points) and the smallest to hold-and-re-evaluate (+0.5). Cost is roughly $0.02 per question. Limitations include a single 1887 French benchmark co-authored by one author of this paper, no significance testing, modest accepted-set precision (0.393), and unmatched compute budgets against baselines.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Historical archival retrieval should be evaluated as source discovery on retrieval metrics, not end-to-end generation metrics.
- TRACE, a training-free orchestration of off-the-shelf components, achieves R@10 = 0.856 and MRR = 0.653 on HistoriQA-ThirdRepublic, beating sparse, dense, graph-based and agentic baselines.
- Gains concentrate on multi-hop and cross-corpus questions (comparative type: 0.674 vs 0.302 for dense retrieval).
- The cumulative ablation attributes the largest gain to corpus-targeted question decomposition (+8.9 points R@10).
- The system costs approximately $0.02 per question under hosted DeepSeek-V4-Flash inference.
Strengths
Retrieval-first evaluation reframing
Evaluates source discovery with R@k/MRR plus accepted-set Pacc/Racc (§4.2), exposing a blind spot in generation-centric agentic RAG evaluation; usefully shows graph systems mostly replicate dense rankings (7.1/6.7 of top-10 shared with dense; net +12/−19 gold documents of ~1,900).
Informative error attribution
§5.1 attributes 82.6% of missed gold documents to the agent loop (vs 17.4% ranked below rank 10) and splits loop misses into judged-irrelevant (36.8%), never-retrieved (36.5%) and held-not-promoted (9.3%), localising relevance judgment — not retrieval coverage — as the bottleneck.
Complete cumulative ablation with per-type breakdown
Deployment-grade cost and latency reporting
Transparent deployment framing
Weaknesses
No statistical uncertainty reporting
Compute budgets unmatched against agentic baselines
Benchmark not independent of the authors
Accepted-set precision omitted from headline claims
Agentic baseline results implausibly weak
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TRACE (Training-free Retrieval with Agentic Corpus Exploration) is an applied source-discovery system for OCR-degraded historical archives, built within the ANR-funded DECIDON project (Third Republic parliamentary debates vs. press) and evaluated on retrieval metrics rather than answer generation. Three claims are advanced: (1) archival retrieval should be measured as source discovery, not generation; (2) an orchestration of off-the-shelf components — planner decomposition, an RRF warm start over BM25/dense/temporal channels, an accept/reject/hold agent loop with deferred re-evaluation, briefing/replanning, and count-grouped reranking over ≤120-character rationales (§3) — outperforms sparse, dense, graph-based and agentic baselines; (3) the whole costs ~$0.02 per question. The title-page note states acceptance at CIKM '26. The reviewed arXiv text is complete: all three result tables are present; Figure 1 is not analysable from text.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TRACE?
TRACE (Training-free Retrieval with Agentic Corpus Exploration) is a system that answers historical research questions by finding citable source documents in digitised archives. An LLM agent decomposes a question into sub-questions, searches multiple retrieval channels (BM25, dense embeddings, dates), explicitly judges each retrieved passage as accept/reject/hold, and produces a final ranked document list — with no model training involved.
How well does TRACE perform, and on what data?
On HistoriQA-ThirdRepublic — 1,752 French questions about 3,386 OCR-degraded parliamentary and newspaper documents from the year 1887 (BnF digitised collections) — TRACE reaches Recall@10 of 0.856 and MRR of 0.653, above BM25 (0.643), dense retrieval (0.734), graph-based HippoRAG 2 (0.737) and LinearRAG (0.727), and agentic baselines A-RAG (0.385) and MA-RAG (0.586).
How much does TRACE cost to run?
About $0.02 per question with the DeepSeek-V4-Flash backbone — roughly $36 for the full 1,752-question benchmark (177.6M input and 39.7M output tokens). Most wall-clock time waits on the hosted model: about 169 seconds per question serially, or about 22 seconds when eight questions run concurrently.
Is the code or the benchmark publicly available?
Yes. The authors release an open-source version at github.com/Kepler1908/TRACE, including corpus conversion and baseline adaptation scripts. The HistoriQA-ThirdRepublic benchmark was published at LREC 2026 and is available via HAL (hal.science/hal-05438255).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (pipeline overview) cannot be analysed from the extracted text; architecture details rely on §3 prose.
- All numbers are quoted from text-extracted tables; minor extraction artefacts in table figures cannot be fully ruled out.
- The paper states acceptance at CIKM '26; any camera-ready differences from this arXiv v1 are not visible here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.