SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding
Shenxi Wu, Yuhong Liu, Haosong Zhang, Tongjin Zou, Yanxun Zhang, Gaochang Chen, Dun Liang, Jiaqi Wang, et al.
cs.AI
Quick verdict
Claims workflow-centered sci-doc benchmark (124 Qs→496 instances) + 15K SFT/8K RL data; sound protocol but small N, no CIs; training eval unreported in §5.
Summary
The paper introduces SciDocBench, a workflow-centered benchmark of 124 expert-authored, difficulty-screened questions for scientific document understanding, spanning five domains, seven capability groups, and 19 subtasks. Each question is instantiated under four matched conditions — English or Chinese questions paired with page-image ('All Images First') versus interleaved-markdown document representations — yielding 496 evaluation instances scored by rule-based, LLM-as-a-judge, or execution-based evaluators. Fourteen multimodal models are evaluated: Claude-Opus-5 leads at 62.60/100, GPT-5.6-Sol follows at 61.00, and no model leads more than two capability groups; document perception (group mean 45.92) and information extraction (50.59) are weakest, and for 11 of 14 models the document-representation gap exceeds the question-language gap. To turn diagnostics into training signal, the authors propose SciDocIR, a block-level representation aligning parsed PDF pages with LaTeX semantics, and build SciDocDataset (~15K supervised fine-tuning and ~8K reinforcement-learning samples across 14 verifiable subtasks) via seeded generation and controlled block-level re-rendering. Strengths include a documented two-stage authoring-and-review protocol with SHA-256-tracked source PDFs and a quantified judge-reliability audit (92.0% three-level agreement, Krippendorff's α=0.854). Key limitations: the effective question count is small (124, so the 496 instances are non-independent), leaderboard gaps such as 62.60 versus 61.00 carry no confidence intervals or significance tests, binary judge agreement is weaker (κ=0.355), and no experiments training on SciDocDataset are reported in the paper's main experiments section. Code is released on GitHub.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No evaluated multimodal model exceeds 62.60/100 on workflow-centered scientific document understanding, with document perception (mean 45.92) and information extraction (50.59) the weakest capability groups.
- Capability leadership is fragmented: no model leads more than two of the seven capability groups, and overall scores conceal large per-model spreads (14.58–31.62 points).
- Document representation matters more than question language for 11 of 14 models, and the two axes interact for some systems.
- SciDocIR enables construction of ~15K SFT and ~8K RL samples across 14 subtasks whose ground truth is deterministic or locked rather than model-generated, forming an evaluation-to-training framework.
Strengths
Matched dual-axis design
Every question runs in four settings (EN/ZH × All-Images-First vs Markdown-Interleaved) with fixed task semantics, yielding controlled findings: 11/14 models show larger representation than language gaps, with interactions (e.g., Gemini-3.6-Flash flips representation preference across languages, §5.4).
Auditable annotation protocol
Appendix C specifies two interfaces with separated author/reviewer roles, reviewer power to reverse correctness labels, immutable submission records, and SHA-256 digests pinning source PDFs against post-screening substitution.
Quantified judge reliability
Ground truth kept out of the generator
Weaknesses
No uncertainty quantification on the leaderboard
Difficulty screening is circular and single-run
Judge noise on pass/fail decisions
Training loop asserted, not demonstrated
Synthetic share of training data
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SciDocBench (arXiv 2609.05141) is a benchmark-plus-dataset paper from CUHK, Shanghai AI Laboratory, and collaborators. It claims three contributions: (1) a workflow-centered benchmark of 124 expert-authored, difficulty-screened questions across five domains, seven capability groups, and 19 subtasks, each instantiated in four matched settings (EN/ZH × All-Images-First page images vs Markdown-Interleaved), i.e., 496 instances; (2) a systematic evaluation of 14 multimodal models; and (3) an evaluation-to-training bridge via SciDocIR (block-level representation aligning visual parsing with LaTeX semantics) and SciDocDataset (~15K SFT + ~8K RL samples over 14 verifiable subtasks). The headline result: Claude-Opus-5 tops the leaderboard at 62.60/100 (Table 2), with GPT-5.6-Sol at 61.00 and Gemini-3.6-Flash at 59.87; scores span 22.44 points down to MiMo-V2.5 at 40.16.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SciDocBench and what does it measure?
SciDocBench is a benchmark for evaluating how well multimodal AI models handle realistic scientific-reading workflows. It contains 124 expert-written questions across five scientific domains and seven capability groups (document perception, information extraction, evidence verification, cross-document reasoning, reconstruction/execution, paper–code alignment, dataset understanding). Each question is tested in four matched variants — English/Chinese × page images vs interleaved markdown — giving 496 evaluation instances scored by rules, an LLM judge, or code execution.
Which AI model performs best on SciDocBench?
In the paper's evaluation of 14 multimodal models, Claude-Opus-5 scores highest at 62.60/100, followed by GPT-5.6-Sol (61.00) and Gemini-3.6-Flash (59.87). No model leads more than two of the seven capability groups, and document perception (group average 45.92) is the weakest capability overall.
How many questions are in SciDocBench?
124 unique expert-authored questions. These become 496 evaluation instances because each question is paired with two languages (English, Chinese) and two document representations (page images and interleaved markdown). The paper explicitly notes the 496 instances are four matched variants, not 496 independent problems.
What are SciDocIR and SciDocDataset?
SciDocIR is a structured representation of scientific papers recording document metadata, page layout, block-level details (text, figures, tables, equations), their relations, and provenance, by aligning parsed PDF pages with LaTeX source. SciDocDataset is training data built on it: about 15,000 supervised fine-tuning samples and 8,000 reinforcement-learning samples across 14 subtasks whose answers are verifiable by rules, locked values, or execution rather than model judgment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt contains two '[...sampling gap...]' regions: the middle of §3.3–§3.4 (score-aggregation formula, instance-construction details) and parts of the references and appendices (D/E); judgments depending on those regions were avoided.
- Figure contents (e.g., capability-profile plots, Figure 4 dataset statistics) are known only from captions; numeric plot values could not be read.
- The provided metadata lists 8 authors, while the paper header lists 11 (adding Zhecan James Wang, Yuhang Zang, and corresponding author Dahua Lin).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.