DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
Zhuoran Yu, Le Thien Phuc Nguyen, Jaden Park, Xinyi Gu, Zexue He, Soochahn Lee, Rogerio Feris, Yong Jae Lee
cs.AI · cs.CV · cs.LG
Quick verdict
DocHop is a carefully verified synthetic benchmark isolating chart-context multi-hop reasoning; well-engineered, but fully synthetic and template-bound limits ecological validity.
Summary
DocHop is a synthetic benchmark of 2,074 single-page document images testing whether multimodal LLMs can use narrative text to resolve target entities before aggregating numeric evidence across multiple charts. Construction is logic-first: a symbolic reasoning trace (constraint nodes plus AND/OR compositions, depth 2-5) is sampled first, then Gemini-2.5-Pro independently generates constraint-satisfying chart tables and a verbalizing narrative; questions are grounded on a semantic reference label so chart-only lookup fails. Programmatic trace execution verifies answers exactly, supplemented by human review of narratives and renderings. Six task categories (Value Retrieval, Counting, Numeric Reasoning, Ranking, Hypothetical Reasoning, Fact Checking) span 41 question templates, 480 topics, and 7 chart types, with controllable reasoning depth and chart count (k in {2,3,4,6}). Evaluating ~20 proprietary and open-source MLLMs, the best model (GPT-5.2 Reasoning) reaches 62.83% overall versus 93.29% human accuracy; open-source baselines land at roughly 9-24%, and accuracy degrades monotonically as depth or chart count grows. Main limitations, acknowledged by the authors: the documents are synthetic with policy-style prose, values restricted to 20.00-99.99, single-page clean layouts, and template-driven questions, so results diagnose a capability rather than predict real-world document performance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DocHop is the first benchmark to require resolving question targets from narrative constraints before aggregating chart evidence, blocking both text-only and chart-only shortcuts.
- A logic-first, reverse-engineered generation pipeline with programmatic trace execution yields provably correct ground truth across 2,074 instances with controllable depth (2-5) and chart count (2-6).
- There is a large gap between human (93.29%) and best-model (GPT-5.2 Reasoning, 62.83%) accuracy, with open-source models at roughly 9-24%.
- Reasoning-enhanced variants consistently beat their non-reasoning counterparts (e.g., GPT-5.2: 40.36% → 62.83%), and performance degrades monotonically with reasoning depth and chart count.
Strengths
Deterministic, leakage-proof answer verification
Because the symbolic trace and chart tables are both accessible, ground truth is computed by executing the trace programmatically (§3.4), eliminating annotation noise that plagues scraped chart-QA datasets; failing instances are discarded and regenerated.
The label-grounding design genuinely blocks single-hop lookup
Questions reference a semantic label (e.g., 'Platinum Tier Night Service') resolved only from the narrative (§3.3), and the context-generation LLM is denied access to chart values and final entity sets (§3.2), so neither text-only nor chart-only shortcuts can succeed — supported by the ~10-11% question-only guessing baselines in Table 1.
Controlled difficulty axes enable diagnosis, not just leaderboard scoring
Multi-stage quality pipeline with independent text/chart synthesis
Informative per-task error structure
Weaknesses
All values live in a narrow 20.00-99.99 band
Narratives may telegraph the target set more than real documents do
Template-driven tasks invite format gaming; Fact Checking is degenerate
Single-seed runs; no variance, CIs, or significance tests
Missing analysis: how much of the gap is chart perception vs. reasoning?
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
DocHop: Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the DocHop benchmark?
DocHop (arXiv 2609.02059) is a benchmark of 2,074 synthetic document-page images, each combining narrative text with 2-6 charts. Models must resolve which entities a semantic label refers to from the text before reading values out of multiple charts, across six task types such as counting, ranking, and fact checking.
How do models perform on DocHop?
The best model, GPT-5.2 in reasoning mode, scores 62.83% overall versus 93.29% for human annotators. Reasoning variants beat their base versions (GPT-5.2: 40.36% vs 62.83%), while open-source models mostly score 9-24%, and accuracy drops steadily as the number of reasoning hops or charts grows.
Why can't models just look at the charts to answer DocHop questions?
Questions never name the target entities directly; they reference a label (e.g., 'Platinum Tier Night Service') that only the narrative defines. The model must first work out which entities qualify under the text's constraints, then find and aggregate those entities' values across several charts.
Is DocHop made from real documents?
No. It is fully synthetic: a randomly generated logical reasoning specification is created first, an LLM (Gemini-2.5-Pro) then writes constraint-satisfying chart data and a policy-style narrative, and answers are verified by programmatically executing the logic against the data. The authors position it as a controlled diagnostic, not a proxy for real-world documents.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Appendix B (example trace figures) and Appendix D's full template pages are rendered as figures with only captions extracted; per-template coverage counts beyond '41 templates' cannot be checked from the text.
- Appendix F states a maximum generation length of '214 tokens', which reads as an extraction artifact or typo for ~2k; judged as stated but flagged.
- Some figure content (Figure 1 arrows, Figure 5 axis values) is described only via captions and surrounding prose; exact per-depth accuracy values in Figure 5 were not numerically extractable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.