Lit3R: Retrieve-Relate-Read for Evidence-Grounded Question Answering over Scientific Literature
Akira Ise, Kotaro Kumagai, Yuta Yamaguchi, Hisanori Ozaki, Yukio Uematsu, Ikuya Yamada
cs.CL
Quick verdict
Training-free retrieve–relate–read system ranks 4th on LitTraceQA; well-engineered with a strong expansion ablation, but evaluation is thin and small-sample.
Summary
This paper presents Lit3R, tus-nlp's system for the LitTraceQA shared task on literature-grounded question answering, which ranked 4th on the official test-set leaderboard (71 questions). The system is a training-free pipeline assembled from off-the-shelf components. The retriever runs up to three iterative rounds: GPT-5.4 decomposes the question into subqueries; hybrid retrieval combines chunk- and paper-level BM25 with a Qwen3-Embedding-8B dense index via reciprocal rank fusion; pseudo-relevance feedback and Qwen3-Reranker-8B reranking follow; an LLM verifier decides whether evidence suffices. A Relate step adds paper-to-paper expansion (SPECTER2 similarity, bibliographic coupling, BM25 over seed papers) fused with the question-anchored ranking. The reader first judges each candidate paper locally, then synthesizes cross-paper evidence into the final answer with evidence locators. On 55 validation questions, recall@50 is 0.968, and paper-to-paper expansion raises multi-paper recall@50 from 0.647 to 0.940. On the test set the system reaches paper F1 0.992, evidence F1 0.737, and perfect multiple-choice accuracy, but table answering is weak (cell accuracy about 0.31-0.33). Limitations: the same 55 validation questions serve both configuration tuning and analysis; components beyond the expansion step are not ablated; no comparison with other leaderboard systems and no error analysis or uncertainty estimates are provided.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A training-free pipeline of off-the-shelf retrieval, reranking, and LLM components is competitive on LitTraceQA, ranking 4th on the official test leaderboard.
- Paper-to-paper expansion (the Relate step) raises validation multi-paper recall@50 from 0.647 to 0.940 and overall recall@50 from 0.814 to 0.968.
- The retriever achieves near-perfect paper retrieval on the test set (macro F1 0.992), with evidence-grounding F1 of 0.737.
- Iterative LLM-driven query decomposition with evidence verification suffices without any task-specific training, given only 55 labeled validation questions.
Strengths
Clean, well-motivated core ablation
Table 2 isolates Ranking A vs the fused A+B system on the same validation set: multi-paper recall@50 rises 0.647 to 0.940 (overall 0.814 to 0.968), directly supporting the Relate step's design rationale laid out in §2.1 and §3.1.
Unusual reproducibility hygiene for a system paper
All retriever prompts and the Stage 1 reader policy are printed in Appendix C with a script claimed to reproduce exact prompts, model versions and the MinerU weight snapshot are pinned (Appendix B, Table 5), and code is released at github.com/tus-ist-nlp/littraceqa.
Careful retrieval engineering with justified design choices
Strong retrieval and selection headroom
Honest, precisely-scoped limitations
Weaknesses
Single-ablation evaluation
No uncertainty or significance reporting
Tuning and analysis share the same 55 questions
Table-answering failure unexplained
Leaderboard context absent
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a shared-task system description for LitTraceQA (GroundLM 2026), a literature-grounded QA benchmark requiring paper retrieval, evidence identification, and answer generation. The authors claim: (1) a training-free pipeline of off-the-shelf components is competitive, placing 4th on the official test leaderboard (71 questions); (2) paper-to-paper expansion (Relate) recovers gold papers that question-anchored retrieval misses, raising multi-paper validation recall@50 from 0.647 to 0.940; (3) the two-stage reader (local reading, then cross-paper synthesis) produces grounded answers, with test evidence F1 of 0.737 and paper F1 of 0.992. Claims are appropriately scoped - the paper explicitly limits itself to LitTraceQA.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Lit3R?
Lit3R is a question-answering system for scientific papers built by the tus-nlp team for the LitTraceQA shared task. It retrieves relevant papers, identifies supporting evidence, and generates grounded answers using entirely off-the-shelf components - BM25, Qwen3 dense embeddings, a cross-encoder reranker, and the GPT-5.4 LLM - with no task-specific training.
How well did Lit3R do in the LitTraceQA shared task?
It ranked 4th on the official test leaderboard of 71 questions. Paper retrieval was nearly perfect (macro F1 0.992) and multiple-choice answers were all correct, but evidence grounding (F1 0.737) and especially table answering (cell accuracy about 0.31-0.33) lagged behind.
What is the Relate step and why does it matter?
Relate expands the candidate-paper list from already-retrieved seed papers using three signals: SPECTER2 embedding similarity, bibliographic coupling (shared references), and BM25 over the seed paper's own text. It lifted validation multi-paper recall@50 from 0.647 to 0.940, recovering gold papers that are lexically and semantically far from the question itself.
Is the code and data available?
Code, prompts, and configuration are released at github.com/tus-ist-nlp/littraceqa. Evaluation data comes from the LitTraceQA shared task (55 validation and 71 test questions); the 27,487-paper corpus of 2024-2025 ML/CV/NLP papers must be rebuilt with MinerU or obtained via the task.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (architecture diagram) is only partially recoverable from extracted text; the pipeline description relies on the prose of §2.
- The Stage 2 reader policy is summarized in Appendix C but its full prompt text is omitted from the extract (stated as released in code), so reader-side behavior could not be fully audited.
- Tables 1-3 were extracted inline; row/column alignment was cross-checked against prose, but minor extraction ambiguity is possible.
- The released repository was not inspected; reproducibility assessments rest on the paper's descriptions alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.