An Empirical Study of VLM Pipelines for Long-Document QA
Kenan E. Ak, Jay Mohta, Gwang Gook Lee, Yan Xu, Dimitrios Dimitriadis
cs.CL · cs.AI · cs.CV · cs.IR
Quick verdict
Rigorous study of modality, retrieval, and agentic access in long-doc QA; sound stats, but the agent's headline lead holds only without thinking mode.
Summary
This empirical study asks how three deployment choices shape vision-language model (VLM) performance on long-document QA: input modality (raw PDF, all page images, extracted text, or top-k retrieved page images), page retriever (BM25, BGE-M3, ColQwen2.5, Nemotron-ColEmbed, and rerank variants), and static versus agentic access via a six-tool function-calling agent. The authors evaluate all configurations on MMLongBench-Doc (1,082 questions) and LongDocURL (2,325 questions) under each benchmark's official scorer, with one shared answer extractor and matched greedy, no-think decoding, using Claude Sonnet 4.5 and locally served Qwen3.5-4B/9B/27B. The Sonnet agent is the top official-scorer configuration on both benchmarks (0.625 and 0.659), leading same-reader top-5 retrieval by +11.7pp on MMLongBench-Doc but only tying the best static pipeline on LongDocURL. Agentic gains emerge only above roughly 27B parameters. Image retrievers dominate text ones (up to 91.4 vs 82.4 Hit@5), and a single Cohere cross-encoder rerank statistically matches the heavier SIRA pipeline. Top-k image retrieval is the most token-efficient mode (~1/7 to 1/4 of all-pages tokens). A ~13pp per-question oracle headroom resists evidence-type routing. Limitations: the agent was never run with thinking mode (static modes gain 4-6pp from it), no code release is stated, open-weight evidence comes from one model family, and external comparisons (DocLens, MDocAgent) use their own protocols.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A six-tool function-calling agent with Claude Sonnet 4.5 is the highest-scoring configuration under the official scorers on both benchmarks (0.625 MMLongBench-Doc, 0.659 LongDocURL), beating same-reader top-5 retrieval by +11.7pp on the former and tying the best static pipeline on the latter.
- Agentic access pays off only above a model-scale threshold of roughly 27B parameters on MMLongBench-Doc: at 4B and 9B the agent trails static page input.
- Retrieval modality dominates retriever choice: image retrievers reach 87.2-91.4 Hit@5 versus at most 82.4 for any text pipeline, and a single Cohere cross-encoder rerank statistically matches the multi-stage LLM pipeline SIRA.
- Top-k image retrieval is the most token-efficient input mode at every reader, at roughly one-seventh to one-quarter of the tokens of sending every page.
- Three strong pipelines leave ~13pp of per-question oracle headroom, but gold evidence-type routing recovers under one point and an observable-feature router performs worse than always running the agent.
Strengths
Single controlled harness across three axes
All configurations share one extractor (Claude Haiku 4.5), matched greedy no-think decoding, DPI 144, and each benchmark's official deterministic scorer (§3), producing a directly comparable leaderboard (Table 1) instead of the cross-protocol comparisons typical of this literature.
Exemplary scoring-bias analysis
Table 15 shows the shared extractor lifts four baselines by +47.1 to +49.7pp (raw prose scores ~0 under string matching) while costing the agent only 0.7pp — demonstrating the protocol favors baselines and that the agent's margin, if anything, is understated.
Confound controls for the headline claim
Honest statistical reporting
Actionable negative result on routing
Weaknesses
Headline claim is decoder-setting-conditional
Cross-system comparisons are not like-for-like
No code or artifact release stated
Single open-weight model family
Failure-scoring and loop-implementation artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Review: An Empirical Study of VLM Pipelines for Long-Document QA (arXiv 2609.29933)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the best way to feed a long PDF to a VLM for question answering?
According to this study, with a frontier model (Claude Sonnet 4.5) a six-tool agentic pipeline that searches, reads pages, and fetches figures/tables scored highest (0.625 on MMLongBench-Doc, 0.659 on LongDocURL under official scorers). With smaller open-weight models (around 9B and below), simply sending top-5 retrieved page images was the best accuracy-per-token default. Sending the raw PDF was the worst value: most expensive and it collapses on documents over roughly 100 pages due to API request limits.
Do image-based retrievers beat text retrievers for finding relevant document pages?
Yes, substantially in this paper: vision late-interaction retrievers (ColQwen2.5 at 87.2 and Nemotron-ColEmbed-3B at 91.4 average Hit@5 across the two benchmarks) beat the best text pipeline (BM25/BGE-M3 with Cohere rerank, about 82) by roughly 9 points, and even a single off-the-shelf cross-encoder rerank matched a much heavier multi-stage LLM retrieval pipeline (SIRA).
Is an agentic pipeline always better than a static retrieval-and-read pipeline for document QA?
No. The paper finds agentic tool use only pays off once the answering model is large enough — roughly 27B parameters and above on MMLongBench-Doc. At 4B and 9B, the agent trailed simply showing all pages, and even at 27B with matched token budgets the gap was within noise. On LongDocURL the agent led at every scale, but only significantly over some baselines.
How much accuracy is left on the table by picking one pipeline instead of the best pipeline per question?
A hindsight oracle picking the best of three strong pipelines per question gains about 13 points over the best single pipeline, showing the pipelines fail on different questions. However, routing on gold evidence type recovered under one point, and a keyword-based router was actually worse than always running the agent, so capturing the headroom needs a finer per-question signal than the authors tested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 appear only as embedded axis-label text fragments; visual trends were judged from captions and the corresponding appendix tables (7, 8, 12).
- arXiv v1 preprint (Sep 2026), not yet peer reviewed; several cited comparisons (SIRA, Nem-CE, DocLens, Scout-era systems) are themselves very recent or concurrent work.
- Extraction is complete through Appendix P, but some appendix table footnotes are interleaved with body text; a few CI values were cross-checked against §M prose.
- Qwen3.5-27B has missing cells (LongDocURL all-images/text and one token count), which limits some cross-reader comparisons exactly where the scale-threshold claim lives.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.