STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation
Vineet Kumar, Meghanadh Pulivarthi, vishwajeet kumar, Jaydeep Sen, Riyaz Ahmad Bhat, Sachindra Joshi
cs.AI
Quick verdict
STAIR makes a book's ToC the retrieval unit of a generative retriever, beating DSI (82.6 vs 76.9 R@1); synthetic queries and constrained decoding temper claims.
Summary
The paper introduces STAIR, a retrieval system that uses a document's table of contents (ToC) as the retrieval unit instead of fixed-length chunks. Built on Mistral Instruct v0.2 and fine-tuned with LoRA, STAIR takes a query plus the full ToC as input and generates the leaf section predicted to contain the answer, with inference constrained to valid ToC leaves. The authors also release SearchTome, a benchmark of 18 open textbooks across six domains (education, finance, law, medicine, natural sciences, social sciences) with ToC-mapped sections and synthetic questions generated per paragraph by Mixtral 8x7B; the source paragraph's leaf serves as the gold label. On SearchTome, STAIR reaches Recall@1 of 82.6%, versus 76.9% for a fine-tuned DSI, 68.7% for out-of-the-box DPR (NV-Embed-v2), 59.5% for BM25, and 13.8% for un-tuned Mistral; domain-level randomization tests report significance at p<0.05 for all six domains. Ablations indicate a near-zero non-leaf ('hallucination') rate for STAIR and larger gains on leaf nodes with few training examples. Key limitations: all queries are LLM-generated with gold labels fixed by construction; the low-hallucination result partly reflects constrained decoding; the DSI baseline uses a 512-token input cap versus 14k for STAIR; and structure-augmented chunk baselines and RAPTOR are not compared.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Using the ToC as both retrieval unit and model input lets an LLM-based retriever beat an equivalently fine-tuned DSI by 5.7 Recall@1 points (82.6 vs 76.9) on SearchTome (Table 4).
- STAIR's non-leaf ('hallucination') output rate stays below 0.05%, far lower than DSI's, because ToC constrains what the model generates (abstract, §6.1).
- ToC in the input enables generalization to leaf nodes with few training examples, where DSI's recall and hallucination degrade (§6.1, Figs 3-4).
- SearchTome is the first benchmark providing a clean ToC-structured view for evaluating long-context retrieval (§3.1).
Strengths
Novel retrieval unit with a controlled comparison
ToC leaves as retrieval units and ToC-in-input fine-tuning (§4.2); DSI and STAIR share the same base model (Mistral v0.2), LoRA setup and training data, so the input difference (ToC present vs absent) is the main delta behind the 5.7-point R@1 gap (Table 4).
New public multi-domain benchmark
SearchTome (§3, Table 1): 18 open textbooks across 6 domains, 70-429 ToC leaves per book, roughly 62k train and ~103k test synthetic questions with gold leaf labels; an anonymized code+data repository is linked in the abstract.
Above-average statistical hygiene
Ablations test the core hypothesis directly
Transparent error accounting and qualitative analysis
Weaknesses
Hallucination claim is partly circular
Fully synthetic evaluation with constructive gold labels
Baseline asymmetries and missing baselines
No run-level variance
Narrow corpus provenance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
STAIR (IBM Research; arXiv 2609.03874, cs.AI) targets a specific weakness of chunk-based RAG: fixed-length chunking discards a corpus's global semantic structure. The proposed remedy makes a book's Table of Contents (ToC) both the retrieval unit and part of the model input for a parametric, DSI-style retriever. The paper makes four testable claims: (1) ToC-augmented fine-tuning lifts Recall@1 to 82.6% on the new SearchTome benchmark versus 76.9% for an equivalently fine-tuned DSI; (2) STAIR's non-leaf ('hallucination') rate stays below 0.05%; (3) ToC in the input yields much better recall on leaf nodes with few training examples (§6.1, Figs 3-4); (4) SearchTome is the first benchmark offering a clean ToC-structured view for long-context retrieval evaluation (§3.1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is STAIR?
STAIR is a retrieval system that fine-tunes an LLM (Mistral Instruct v0.2 with LoRA) to read a user query together with a book's complete table of contents and directly generate the ToC leaf section expected to answer the query, rather than retrieving fixed-length text chunks.
What is the SearchTome benchmark?
SearchTome is a benchmark of 18 openly licensed textbooks across six domains (education, finance, law, medicine, natural sciences, social sciences), each parsed into a table of contents mapped to section text, with synthetic train/dev/test questions whose gold label is the ToC leaf containing the source paragraph.
How much better is STAIR than existing retrievers?
On SearchTome, STAIR's Recall@1 is 82.6%, versus 76.9% for a fine-tuned DSI, 68.7% for out-of-the-box DPR (NV-Embed-v2), 59.5% for BM25, and 13.8% for un-tuned Mistral; the DSI-vs-STAIR gap is statistically significant in all six domains (randomization test, p<0.05).
Is the code and data for STAIR available?
Yes - the paper links an anonymized 4open.science repository said to contain the benchmark data and training code, and the 18 source textbooks are publicly available via links in the paper's Table 1.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are image content; only their captions and in-text descriptions are present in the extracted text, so figure-internal values (e.g., exact per-leaf curves in Figs 3-4) could not be independently checked.
- The linked anonymized 4open.science repository was not accessed; availability and reproducibility statements rest on the paper's text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.