PageRecall: Measuring Page Selection in Literature-Grounded Question Answering
cs.IR · cs.CL
Quick verdict
Shows evidence grounding is gated by page selection (52.6% recall), not reading (94%); pragmatic whole-paper fix; honest but small-sample shared-task report.
Summary
This system paper describes the Everest entry to LitTraceQA (GroundLM @ EMNLP 2026), which requires retrieving papers from a 27,487-paper pool, citing page-level evidence (page plus table/figure identifier), and answering in a requested format. The central contribution is a controlled decomposition on 97 development (question, paper) units: the BM25 page selector surfaced the gold page in only 52.6% of units, while the reading model, conditional on seeing it, cited correctly in 45 of 48 locators (94%); when the gold page was absent, the reader abstained in only 14 of 45 cases, so silent failures dominated. Because median papers fit in context (~19k tokens), the fix is to show the whole document (100% gold-page recall), with deterministic anchors resolving ordinal questions such as 'the 24th reference'. The paper candidly reports a development-fitted padding heuristic that cost 0.245 paper F1 on test, a positional bias toward the final multiple-choice option corrected by permutation voting (0.840 to 0.920 accuracy, on only six corrections), and the negative result that entailment-gate groundedness does not predict correctness. Final test scores: 0.762 paper F1, 0.441 evidence F1, 0.577 composite. Limitations include small splits (55/71 questions), a closed model without seed control, and no visual page-retrieval baseline.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Evidence grounding in this pipeline is limited by page selection, not reading: the selector showed the gold page 52.6% of the time while the reader cited correctly in 45 of 48 locators (94%) given that page.
- When the gold page was absent, the pipeline failed quietly about twice as often as visibly: 24 wrong-page emissions versus 14 abstentions across 45 units.
- Showing the whole paper achieves 100% gold-page recall on parseable papers because the median paper (~19k tokens) fits in context, making page ranking optional at this document length.
- A development-fitted padding heuristic (pad multi-paper clusters to four) cost 0.245 paper F1 on test and was mildly harmful even on its own split, arguing for individual ablation over bundled evaluation.
- Entailment-gate groundedness verdicts are close to independent of answer correctness, and the evaluation metric structurally cannot reward abstention over guessing.
Strengths
Clean experimental decomposition isolating the selector
Section 5 evaluates (question, paper) units kept only where stage-1 retrieval succeeded and a parse exists, separating selector recall (52.6%) from conditional reader accuracy (45/48, 94%); the measurement involves no model calls and is deterministic and repeatable.
Candid failure reporting, including self-inflicted errors
Section 7 shows the dev-fitted pad-to-4 heuristic was mildly harmful even on dev (F1 sweep 0.840/0.831/0.814/0.825) yet cost 0.245 paper F1 on test; Section 9 reports the entailment gate as a negative result (unsupported MC answers 15/16 correct vs supported 22/25).
Strong reproducibility engineering
Well-argued, economics-grounded design pivot
Metric-aware diagnosis of answer-level pathologies
Weaknesses
Headline conditional estimates rest on very small counts
The 'retrieval, not reading' framing overstates a partial result
Selector comparison table is not like-for-like and omits visual retrieval
Whole-paper fix untested against its acknowledged main threat
Reproduction requires a released response cache, not re-running
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of the PageRecall paper?
In a scientific question-answering pipeline, the step that selects which PDF pages the language model sees was the bottleneck: it surfaced the correct (annotator-designated) page only 52.6% of the time, while the model that reads pages cited the correct page 94% of the time (45 of 48 cases) when it was actually shown. The authors conclude grounding failures came from retrieval, not reading.
Why does the system show the whole paper instead of selecting pages?
Because the papers in this benchmark are short enough to fit in the model's context (about 19,000 tokens at the median), showing every page raises gold-page recall from 52.6% to 100% at essentially the same cost. Page ranking is kept only as a fallback for papers too long to fit, which never occurred on the test split.
What scores did the system achieve on the LitTraceQA test split?
On the hidden test split, evaluated by the organizers' online evaluator: 0.762 paper F1, 0.441 evidence F1, 0.920 multiple-choice accuracy, and a composite of 0.577 (the ranking metric). Table answering was the weakest component, with 0.242 macro cell accuracy.
Is the code and data for this paper available?
Yes. Code, committed predictions, run artifacts, and a reproduce.py harness are released at github.com/aadityc91/littraceqa-groundlm2026. Because the system uses a closed model (claude-sonnet-5) with no seed control, exact reproduction requires the separately released response cache; the harness verifies the paper's claims against these artifacts and reports skips rather than silent successes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text appears complete (abstract through references) with only minor artifacts (e.g., one garbled inline citation 'DasigiarXiv'); no sections are visibly missing.
- Figure 1 (pipeline diagram) is available only as extracted text layout; its visual rendering was not assessed.
- Test-split scores are taken from the organizers' online evaluator as reported by the authors; independent verification was not possible from the text alone.
- The benchmark and shared-task papers it cites (Liu et al., 2026; Wang et al., 2026a,b) could not be checked against this text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.