ICDAR2026 Competition on Multimodal Reasoning over Documents in Multiple Domains
Artemis Llabrés, Marc Serra Ortega, Tomàs Ockier, Samuel Ortega Cuadra, Amritpal Singh, Christos Georgakilas, Andrey Barsky, Ernest Valveny, et al.
cs.CL
Quick verdict
ICDAR2026 DocVQA2026 report: 8-domain document VQA benchmark; structured pipelines beat zero-shot VLMs, but 160-question test set lacks statistical power.
Summary
This competition report introduces DocVQA2026, a visual question answering benchmark spanning eight document domains — business reports, scientific papers, slides, posters, maps, comics, infographics, and engineering drawings — motivated by saturation of earlier DocVQA-style benchmarks (reported near 98% on DocVQA, 91% on InfographicsVQA). Questions require spatial, temporal, and multi-hop reasoning rather than extraction; about 6.25% of validation questions are intentionally unanswerable. The organizers ran the challenge on the RRC platform with deterministic evaluation — strict matching for numbers, dates, and versions, then relaxed ANLS matching at a 0.9 threshold — explicitly avoiding LLM-as-a-judge. Twenty valid submissions from eight teams competed in three parameter bands. The >35B winner, Luna+IDA with a multi-agent ensemble plus fine-tuning, reached 60.00% accuracy; the 8–35B band tied at 35.63% between ARGUS (27B, OCR+retrieval) and Perceive-Reason-Code (27B, code-execution agent); the ≤8B winner scored 18.75%. Zero-shot Gemini-3.1-Pro-preview managed 37.50%, and a 9B engineered system (HALLEY, 20.00%) beat GPT-5-mini (17.50%) and 32B zero-shot Qwen (15.63%). The main weaknesses are statistical: the test set holds only 160 questions (20 per domain), no significance tests or confidence intervals are reported, and single-run temperature-1.0 baselines make close rankings fragile.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DocVQA2026 is a far-from-saturated, eight-domain document VQA benchmark whose questions require spatial, temporal, and multi-hop reasoning rather than extraction.
- The strongest competition systems rely on structured evidence extraction, retrieval, verification, and multi-component orchestration rather than single-pass prompting.
- Overall accuracy does not fully characterize systems: the two 27B methods tied at 35.63% exhibit opposite domain strengths (ARGUS text-heavy; Perceive-Reason-Code spatial domains).
- Deterministic strict-plus-ANLS(0.9) evaluation avoids the cost and calibration biases of LLM-as-a-judge evaluation.
- Careful system engineering can partly compensate for scale: the 9B HALLEY system (20.00%) outscores GPT-5-mini (17.50%) and 32B zero-shot Qwen variants (15.63%/14.38%).
Strengths
Genuinely diverse benchmark with a well-argued motivation
§1 quantifies saturation of prior benchmarks (~98% DocVQA, ~91% InfographicsVQA, ~88% MP-DocVQA) and §1's related-work paragraph identifies concrete gaps (ComicsPAP is pick-a-panel, MapBench is navigation, MMLongBench-Doc covers few domains) that the 8-domain design of Fig. 2 addresses.
Deterministic evaluation design, well specified
§2.1 defines strict matching (numbers+units, dates, version strings) with an ANLS≥0.9 relaxed fallback including normalization and multi-variant ground truth; the exact shared prompt is published in Fig. 6; the rejection of LLM-as-a-judge is justified by citation [5] on judge miscalibration rather than assertion.
Controlled competition protocol with scale disclosure
Analysis beyond aggregate accuracy
Unusually good continuity for a competition paper
Weaknesses
No statistical treatment of rankings on a tiny test set
Single-run, temperature-1.0 baselines
Band integrity and compute opacity
Representative systems described only narratively
Version naming inconsistencies undermine baseline reproducibility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a competition report, not a research paper: it documents the ICDAR2026 Competition on Multimodal Reasoning over Documents, built on the new DocVQA2026 dataset (public on Hugging Face, VLR-CVC/DocVQA-2026). The benchmark spans eight domains — business reports, scientific papers, slides, posters, maps, comics, infographics, engineering drawings (§2, Fig. 2) — with 80 validation and 160 test questions (10 and 20 per domain) and ~6.25% intentionally unanswerable validation questions. The authors' central claims: (1) existing benchmarks are saturated (~98% DocVQA, ~91% InfographicsVQA, ~88% MP-DocVQA, §1) and too narrow, so a multi-domain reasoning benchmark is needed; (2) the best systems rely on structured evidence extraction, retrieval, verification, and orchestration rather than single-pass prompting (§4, §5); and (3) overall accuracy alone does not characterize systems, since tied methods differ per domain (§4.3, Fig. 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DocVQA2026?
A document visual question answering benchmark introduced with the ICDAR2026 competition, containing 80 validation and 160 test questions across eight document domains (business reports, scientific papers, slides, posters, maps, comics, infographics, engineering drawings). Questions require spatial, temporal, or multi-hop reasoning rather than simple extraction, and about 6.25% of validation questions are intentionally unanswerable to test abstention.
What is the best accuracy achieved on DocVQA2026?
In the competition, the best system — Luna+IDA with a multi-agent ensemble plus fine-tuning, in the >35B parameter band — scored 60.00% test accuracy. The best zero-shot baseline, Gemini-3.1-Pro-preview, scored 37.50%. The authors consider the benchmark far from saturated.
Where can I access the DocVQA2026 dataset and evaluation?
The dataset is publicly available on Hugging Face (VLR-CVC/DocVQA-2026) with validation ground truth included. Test answers are hidden; evaluation runs through the Robust Reading Competition server (rrc.cvc.uab.es, channel 34), which the authors say will remain available together with the evaluation code.
Which document domains are hardest in DocVQA2026?
Maps are the most difficult domain across nearly all submitted methods; engineering drawings and comics are also challenging, especially for zero-shot models. Business reports, science papers, and slides are the strongest domains for the top systems.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 7 and 8 heatmap values appear in the extraction as flat number sequences without grid alignment, so exact per-domain values cannot be reliably attributed; this review cites only Table 1 numbers and qualitative domain statements.
- The Fig. 6 prompt text carries PDF line-break artifacts ('↪→'); its content is otherwise complete.
- Metadata lists 8 authors, but the text's author block includes a ninth (Dimosthenis Karatzas); the metadata is likely incomplete rather than the text.
- Table 1 continues across a page break ('Continued on next page') in the extraction; all rows appear present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.