Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents
Jiaqiang Li, Yajie Yang, Zhiheng Xi, Jiadong Chen, Enyu Zhou, Senjie Jin, Yang Nan, Jiazheng Zhang, et al.
cs.AI
Quick verdict
Sci-MMR's argument-graph benchmark is a genuinely diagnostic, unusually well-audited contribution; interpretation is constrained by 235 tasks, LLM-judged coverage and unquantified variance.
Summary
Sci-MMR is a benchmark for multi-step, evidence-grounded scientific reasoning whose 235 multi-hop tasks are argument graphs linking scientific claims, citation-grounded knowledge, visual evidence, and image regions, drawn from peer-reviewed papers across four disciplines and 35 domains, averaging nine figure panels per task. A human-in-the-loop pipeline (MinerU extraction, LLM graph construction, expert region annotation, deterministic filters, an eight-dimensional LLM rubric, expert review with κ=0.77–0.87) filters 800 papers down to 235 tasks. Eight frontier models are evaluated in four settings: caption-only, direct visual reasoning, gold-evidence hints, and crop-tool agentic use. Answer accuracy consistently exceeds complete-evidence recovery by more than 20 points; gold evidence lifts pooled accuracy 27.4 points (46.1%→73.5%) while the strongest model still reaches only 69.1% on hard tasks; crop tools add just 4.5 pooled points, and only 12.7% of crop-called runs recover all annotated regions. Routing 1,014 errors attributes 57.2% to evidence acquisition and 31.8% to integration. Stated limitations include small scale, LLM-heavy construction and judging, and coverage metrics that measure only explicitly stated evidence.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Answer accuracy consistently exceeds complete-evidence recovery by more than 20 points on Sci-MMR, showing answer-only evaluation overestimates evidence-grounded reasoning (dataset-level).
- Evidence acquisition is the primary bottleneck: 57.2% of 1,014 incorrect Direct Visual Reasoning outputs are evidence-related, dominated by access/localization failures (40.1%).
- Gold-evidence injection improves accuracy by 19.1–37.0 points per model (pooled +27.4), yet the strongest model reaches only 69.1% on hard tasks, so integration remains a distinct bottleneck.
- Active localization helps only modestly: crop tools add +4.5 pooled points, and only 12.7% of crop-called runs recover the complete annotated region set.
Strengths
Four-setting diagnostic protocol cleanly separates acquisition from integration
Section 4 defines Caption-only, Direct Visual Reasoning, Evidence-Hint (gold evidence injected) and Agentic Tool-Use. The gold-evidence intervention (RQ1) shows pooled accuracy rising 27.4 points (46.1%→73.5%, Table 13) while crop tools add only +4.5 (Table 14), separating 'can't find evidence' from 'can't use it'.
Unusually thorough quality control for a benchmark
Deterministic depth filter (removes 54%), 8D LLM rubric (filters 17%), expert review of all 260 handoff tasks (25 rejected) with region IoU≥0.7 acceptance and pre-adjudication κ=0.77–0.87 (Tables 5–6). Judge validation: Gemini 3.5 Flash reaches 92% accuracy agreement with adjudicated human consensus over 384 audited responses, ρ=0.85 for E-Cov (Tables 8–9).
Error attribution with declared routing rules and sensitivity analysis
The argument-graph formalism is precise
Diagnostically informative headline result
Weaknesses
Verbosity/format confound in the headline claim
No uncertainty quantification
'Integration bottleneck' conflates uptake with integration
Judge family overlap
Scale is very small for a benchmark
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Sci-MMR (arXiv 2609.11243v2, Fudan NLP Group) is a benchmark for multi-step, evidence-grounded scientific reasoning. Each of 235 tasks is a directed-acyclic Argument Graph (§3.1) linking visual sources, evidence statements, citation-grounded knowledge, and claims, built from peer-reviewed papers across four disciplines and 35 domains with ~9 figure panels per task. Construction (§3.2, App. A–B) runs 800 papers (2020–2026; Nature/Science/Cell/PNAS) through MinerU extraction, LLM graph construction, expert region annotation, deterministic depth filtering (−54%), an 8D LLM rubric (−17%), and expert review (260 tasks reviewed, 25 rejected; κ=0.77–0.87, Tables 5–6). Central claims: (1) answer accuracy exceeds complete-evidence recovery by >20 points for all eight frontier models tested; (2) evidence acquisition causes 57.2% of failures vs 31.8% for integration; (3) crop-tool agency yields only +4.5 pooled points.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Sci-MMR?
Sci-MMR is a benchmark of 235 multi-hop scientific reasoning tasks, each built as an argument graph linking a scientific claim to citation-grounded knowledge, visual evidence, and the specific image regions that support it. It is designed to test whether AI models can gather and use complete scientific evidence, not just produce a correct-sounding final answer.
What is the headline finding of the Sci-MMR paper?
Across eight frontier multimodal models, answer accuracy consistently exceeds complete-evidence recovery by more than 20 percentage points: models often reach the right conclusion while finding only part of the supporting evidence. Only about 12.7% of model crop attempts recover all annotated evidence regions.
Why do multimodal models fail on Sci-MMR?
The authors' error routing over 1,014 incorrect answers attributes 57.2% of failures to evidence acquisition — mainly locating and assembling evidence scattered across multiple figure panels — and 31.8% to evidence integration, meaning models struggle even when all evidence is handed to them.
How big is Sci-MMR and is the data available?
It contains 235 tasks spanning four scientific disciplines (physics, computer science, biology, chemistry) and 35 domains, with about nine figure panels per task. The provided paper text does not state a public release; anyone wanting to use it should check the authors' repository for the dataset and scoring code.
How are answers scored in Sci-MMR?
A Gemini 3.5 Flash LLM judge checks four binary correctness conditions for the final answer and separately scores evidence coverage (fraction of required evidence nodes substantively expressed) and claim coverage (intermediate claims). On a 384-response audit, the judge agreed with human consensus on 92% of final accuracy labels.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text contains no code/data release statement or URL; release status is therefore unverified (it may appear in a section not included in the sampled excerpt).
- The Caption-only aggregate results table is not present in the provided text (its protocol and per-setting denominators are); it may exist in a sampling gap rather than being absent from the paper.
- No statistical tests, CIs, or seeds/decoding settings are reported in the provided text; they may exist in an unrevealed section, but nothing in what was shown indicates they do.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.