Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA
Nikhil Reddy Pottanigari, Sepideh Kharaghani, Saverio Vadacchino, Alejandro Posada, Ying Zhang
cs.AI
Quick verdict
Claims page images beat text on doc QA accuracy within a 50-page cap at growing latency cost; controlled, honest work, but no significance testing.
Summary
The paper isolates a narrow but consequential choice in document QA: whether the model receives page images, OCR-extracted text, or both. Prompt, GPT-5.2 judge, and scoring pipeline are held fixed while input representation varies across four commercial endpoints (Gemini 3.1 Flash-Lite/Pro, GPT-5.4/-mini), two corpora (an internal enterprise KB with 1,447 documents and 5,169 questions; public MMLongBench-Doc with 135 documents and 1,091 questions), and two regimes (gold evidence pages vs. full document). To separate modality effects from truncation, the main analysis restricts to documents fitting a 50-page-image cap. Within that budget, image-bearing inputs lead on accuracy in every length bucket on both corpora (KB ORACLE averages 0.708–0.731 vs. 0.611 for text), while text latency stays roughly flat and image latency and cost grow with document length (1.5–2.4× cost premium). Text and images answer different questions, with exactly one representation correct on 19–25% of items. A TF-IDF logistic-regression router reading only question text gains 2.6 pp over always-text while cutting median latency ~30% versus always-vision on a document-disjoint split, explicitly framed as a proof of concept. Key limitations: no significance testing, proprietary OCR and internal KB, only closed-weight endpoints, and a single LLM judge without human validation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Within the 50-page image budget, image-bearing inputs (IMAGE or TEXT+IMAGE) lead on accuracy in every document-length bucket on both corpora (Table 3, Figure 2).
- Document length governs the cost of the text–image trade-off, not its direction: text latency stays roughly flat while image latency and cost grow steadily (Tables 4–5, Figure 2).
- Text and images fail on different questions, with exactly one representation correct on 19–25% of items across all reported cells (Table 6).
- A TF-IDF router reading only question text gains 2.6 pp over always-text while cutting median latency ~30% versus always-vision on a document-disjoint holdout (Table 7), as a proof of concept.
- The apparent accuracy crossover favoring text on long documents is an artifact of the 50-image page cap, not a modality property (Appendix C, Figure 3).
Strengths
Genuinely controlled comparison
Prompt, GPT-5.2 judge, decoding (temperature 0, seed 30), and scoring are held fixed across 3 representations × 4 endpoints × 2 corpora × 2 regimes (Section 3), so within-cell accuracy differences attribute to representation alone — rare discipline in the text-vs-image literature the paper reviews.
Truncation vs. modality cleanly separated
The ≤50-page restriction (99.86% of KB docs) plus Appendix C shows the text 'crossover' on long documents is a page-cap artifact: the gain from restricting concentrates in IMAGE/FULL (+0.041 to +0.058 on MMLongBench-Doc, Table 9), an order of magnitude above TEXT cells, and latency drops (−1.0 to −1.6 s, Table 10) land on the same cells.
Honest uncertainty framing
Complementarity result grounds the router
Reproducible public anchor despite proprietary assets
Weaknesses
No statistical inference anywhere
Proprietary text pipeline confounds the central comparison
Narrow, closed model coverage
Router evaluation choices blunt the headline
Judge validity unverified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper isolates one preprocessing decision in document QA — whether the model receives page images (IMAGE, 300-DPI renders), extracted text (TEXT, proprietary OCR), or both (TEXT+IMAGE) — with prompt, GPT-5.2 judge, decoding (temperature 0, seed 30, 3,000-token cap), and scoring held fixed. The grid is 3 representations × 4 commercial endpoints (Gemini 3.1 Flash-Lite/Pro, GPT-5.4/-mini) × 2 corpora (internal enterprise KB: 1,447 docs / 5,169 questions, median 4 pages; public MMLongBench-Doc: 135 docs / 1,091 questions, mean 48.3 pages) × 2 regimes (ORACLE gold evidence pages; FULL document). Four claims are made: image inputs lead within the 50-page budget at every document length; length scales cost and latency, not the accuracy direction; 19–25% of questions are answered correctly by exactly one representation; and a TF-IDF router captures part of the gap. The claims are conservatively scoped and, as far as the extracted tables show, internally consistent with the reported numbers.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Should I send document pages as images or as extracted text to an LLM for question answering?
According to this paper, within a 50-page image budget, page images (or images plus text) gave higher accuracy than OCR text across four commercial models and two corpora, including at every document length tested. The trade-off is cost: images cost 1.5–2.4× more and get slower as documents grow, while text latency stays roughly flat. On documents longer than the image cap, images get truncated and text can win.
What is the TF-IDF router proposed in this paper?
It is a lightweight proof-of-concept classifier: TF-IDF features of the question text feed a logistic regression that decides, per question, whether to send text or page images. On a held-out split with no document overlap, it improved accuracy by 2.6 percentage points over always sending text while cutting median latency about 30% versus always sending images. Training it requires knowing which representation answered each question correctly, so the authors present it as an upper-bound estimate, not a deployable system.
Does this paper show that text beats images for long documents?
No. A crossover favoring text appears only for documents longer than the 50-page image cap, where image inputs are truncated by the provider limit. Within the untruncated range (≤50 pages), image-bearing inputs led on accuracy in every length bucket on both corpora. The authors explicitly attribute the crossover to the fixed deployment budget, not to the modality itself (Appendix C, Figure 3).
Can I reproduce this paper's results on public data?
Only partially. The MMLongBench-Doc half uses a public benchmark, and the paper publishes exact prompts, model identifier strings, the 50-page cap, seeds, and the judge template. However, the enterprise knowledge base (roughly five sixths of the questions), the proprietary OCR pipeline, and all code are not released, and re-running requires paid Gemini and OpenAI API access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are available only as captions in the extracted text; bucket-level claims (e.g., flat text latency, the truncated-range crossover) could not be visually verified.
- The extracted text appears complete (Sections 1–7, Limitations, Ethics, Appendices A–E, Tables 1–12), supporting high confidence; numeric values are quoted as extracted.
- A minor prose/table mismatch was noticed: Section 4.1 says gains 'up to 14.8 pp' on the KB under ORACLE, while Table 3 implies ~15.3 pp for Gemini Flash-Lite IMAGE (0.729) vs TEXT (0.576); the review relies on table values.
- All models, judge, and dates reflect the paper's own stated versions (e.g., GPT-5.4, GPT-5.2 judge, September 2026 preprint); no external verification of endpoint behavior was possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.