Bridging Static and Agentic RAG for Taiwanese Historical Question Answering
Kai-Hsin Chen, Wei-Yu Chen, Xuanjun Chen, Jyh-Shing Roger Jang
cs.CL · cs.IR
Quick verdict
Agentic vs static RAG tie on aggregate yet differ on 70.8% of questions; post-hoc selection recovers 60% of oracle headroom — solid but LLM-judge dependent.
Summary
This paper presents a controlled comparison of static and agentic retrieval-augmented generation (RAG) for Taiwanese historical question answering. Both pipelines share a hybrid retrieval backend (Qwen3-Embedding dense + BM25 sparse, RRF fusion, BGE-reranker-large) and the same generator (Claude Sonnet 5); they differ only in orchestration — a fixed multi-query, top-15 static procedure versus an LLM agent invoking the retriever through an MCP tool with top-5 passages per call. On the 120-question Minqing archive benchmark, scored on four dimensions by an LLM judge (Claude Opus 4.8) against reference answers generated by Claude Opus 5, aggregate composite scores are statistically indistinguishable (4.21 vs 4.12, p=0.65), yet the pipelines differ on 70.8% of questions. An oracle selector would gain +0.2417 composite; a GPT-5.5-based post-hoc pairwise selector comparing responses and cited evidence significantly outperforms either pipeline (+0.1458, p=0.0012), recovering 60.34% of oracle headroom, robust to tie handling. A supplementary judge-free TFCAD evaluation (n=102) shows no significant difference. Behavior analysis links agentic wins to query reformulation. Limitations: LLM-judge evaluation with LLM-generated references, no human validation, two domain-specific benchmarks, a single generator, proprietary model versions, no code release, and doubled inference cost from running both pipelines.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a shared retrieval backend and generator, agentic RAG does not provide a statistically significant aggregate advantage over a well-designed static pipeline on the Minqing benchmark (4.2104 vs 4.1188, p=0.6543).
- The aggregate similarity masks question-level complementarity: the two pipelines differ on 85 of 120 questions (70.83%) with near-even wins (41 vs 44), and per-question oracle selection would add +0.2417 composite over the better pipeline.
- A GPT-5.5 post-hoc selector comparing both responses and their cited evidence significantly outperforms either individual pipeline (+0.1458, p=0.0012), recovering 60.34% of the oracle headroom, robust to tie-handling choices.
- Agentic success is associated with redirecting search via query reformulation (Jaccard 0.076 wins vs 0.141 losses) and with recovering gold evidence missed in the first retrieval round (18/120 questions, +0.54 advantage).
Strengths
Genuinely controlled comparison design
Both pipelines share Qwen3-Embedding/Qdrant dense + BM25/Elasticsearch sparse retrieval, RRF fusion (k=60), BGE-reranker-large reranking, and Claude Sonnet 5 generation (§3.1, §4.2), isolating orchestration — the usual model-vs-retriever confound in agentic-vs-static comparisons is avoided. The closed-book ablation (§6.1: 2.3146 vs +1.8958/+1.8042 retrieval gains) further contextualizes the orchestration choice as second-order.
Quantified complementarity beyond aggregates
The paper does not stop at 'no significant difference' (Table 1, p=0.6543) but decomposes it: 85/120 questions differ (70.83%), wins split 41 vs 44, paired-score correlation only r=0.405 (Fig. 3), oracle headroom +0.2417 — 2.64x the aggregate gap. This reframing is the paper's real conceptual contribution.
Exemplary robustness analysis of the selector
Mechanistic grounding of when adaptive retrieval helps
Transparent statistical reporting
Weaknesses
LLM-judge circularity surrounds all headline numbers
Context-budget and design-choice confounds
Limited generalization scope
No multiplicity control across dimension tests
Reproducibility and cost left open
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: under an identical retrieval backend and generator, does agent-controlled retrieval beat a fixed static procedure, and can question-level disagreement between the two be exploited? On the Minqing benchmark (120 open-ended questions over 46,709 Ming-Qing administrative documents, 117,786 chunks), the authors report: (1) no significant aggregate difference (composite 4.2104 agentic vs 4.1188 static; Δ+0.0917, 95% CI [−0.0854, +0.2687], Wilcoxon p=0.6543; Table 1); (2) disagreement on 85/120 questions (70.83%) with near-even wins (41 vs 44), oracle headroom of +0.2417, and paired-score correlation of only r=0.405 (Fig. 3); (3) a GPT-5.5 post-hoc selector scoring 4.3563 — a significant +0.1458 over the better pipeline (p=0.0012), recovering 60.34% of oracle headroom (Table 2). Claims are scoped honestly; the pre-generation routing suggestion is explicitly framed as unproven future work.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does agentic RAG beat static RAG according to this paper?
Not on aggregate. With the same hybrid retrieval backend and the same generator, composite quality scores on the 120-question Minqing Taiwanese archives benchmark were 4.21 (agentic) vs 4.12 (static), a difference that is not statistically significant (p=0.65). However, the two pipelines disagreed on 70.8% of individual questions with nearly even wins, meaning the better choice depends on the specific question rather than on the paradigm.
What is the paper's post-hoc answer selector and how much does it help?
It is an LLM (GPT-5.5) that receives the original question, both candidate answers, and the union of the evidence each answer cited, then picks the better answer or abstains with a tie (each pair is judged twice with order reversed). It scored 4.36 vs 4.21 for the better single pipeline — a statistically significant +0.15 gain, recovering 60.34% of the gap to an oracle that always picks the better answer. The catch: both pipelines must run first, roughly doubling inference cost.
What datasets and models were used in the study?
Two Taiwanese historical resources: the Ming-Qing Taiwan Government Administrative Archives (46,709 documents split into ~117,800 chunks, with 120 open-ended questions) evaluated by an LLM judge on correctness, completeness, faithfulness, and relevance; and 102 fill-in-the-blank/multiple-choice questions from a Taiwanese Folklore and Cultural Artifacts Dictionary for objective accuracy. Retrieval used Qwen3 embeddings (Qdrant) plus BM25 (Elasticsearch) with BGE reranking; answers were generated by Claude Sonnet 5; the judge was Claude Opus 4.8 using reference answers written by Claude Opus 5.
What are the main limitations of this study?
Answers are scored by an LLM judge against LLM-generated reference answers, with no human validation; only two domain-specific Taiwanese historical benchmarks and a single generator were tested; no code or data release is stated; prompts are only partially provided; and the selector requires running both pipelines, doubling inference cost. The reported 60.34% headroom recovery should therefore be read as conditional on this particular judge configuration.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are present only as embedded text labels in the extraction; visual details of the paired-score scatter and per-dimension bars could not be inspected.
- Appendix B prompts are abridged by the authors ('implementation-specific instructions omitted'), so exact judge and selector behavior cannot be fully verified from the text.
- No code or benchmark availability statement appears in the extracted text; a release link could exist outside the extracted portions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.