Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning
Raj Jaiswal, Sree Krishna Uppalapati, Dhruvkumar Patel, Ria Khatoniar, Tanuja Ganu, Rajiv Ratn Shah
cs.AI
Quick verdict
Five-task diagnostic separating perception from reasoning in multimodal physics/geometry QA; sound design, but no significance tests and confounded GQ ceiling.
Summary
The paper introduces a five-task diagnostic that manipulates visual grounding — question-only (Q), raw image (IQ), model caption (CQ), question-aware caption (CQ*), and human ground-truth caption (GQ) — across three physics benchmarks (PhysReason, PhysUniBench, SeePhys) and two geometry subsets (OlympiadBench 152 problems, Geometry3K 300), evaluating six multimodal LLMs under CoT prompting. A GPT-4V judge labels perception errors (description/explanation/value, AND logic) and reasoning errors (miscomprehension/conceptual/calculation), validated against four human annotators (Cohen's kappa 0.74-0.84). Key results: adding images causes failure on up to 46.1% of problems solved from text alone (Qwen2.5-VL 72B, PhysUniBench); accuracy generally rises from raw images toward human captions; physics perception failures propagate mainly into calculation errors whereas geometry failures become conceptual misapplication; and InternS1-mini underperforms the weakest baseline on every task with 20-63% truncated responses and >95% combined failure on geometry. Main limitations: no statistical testing or confidence intervals, single caption sample per problem, small geometry subsets, and GQ captions that embed step-by-step reasoning hints, confounding the interpretation of the GQ-IQ gap as pure perception burden; the Section 7 claim that GQ is consistently highest is contradicted by the paper's own Table 8.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Adding the diagram causes failure on up to 46.1% of problems that models already solve from text alone (Vision Hurts Rate, Table 3).
- Accuracy generally improves from raw images (IQ) to human-authored captions (GQ), and high recovery under GQ separates perception-blocked failures from genuine reasoning bottlenecks (Table 5).
- The reasoning error that follows a perception failure is domain-determined: physics failures become calculation errors (~40%), geometry failures become conceptual misapplication (~49%), regardless of perception error type (Table 4).
- InternS1-mini, despite scientific pretraining and extended thinking, falls below the weakest standard model on every task, with 20-63% of responses truncating before a final answer and >95% combined failure on geometry (Section 8).
Strengths
Controlled factorial design cleanly separates perception from reasoning
Five input conditions (Q/IQ/CQ/CQ*/GQ, Section 4.3) hold the reasoning prompt fixed while varying visual grounding, enabling attribution such as the Vision Hurts Rate (Table 3) and perception-caused recovery rates (Table 5); the physics-vs-geometry contrast is an informative design axis.
Judge validation with human agreement
GPT-4V judging is validated against four annotators with Cohen's kappa 0.74-0.84 reported per benchmark (Table 6), and all caption-generation and judge prompts are released verbatim in Appendices B-C.
Honest failure accounting for InternS1-mini
Concrete, quantified phenomena
Reproducibility artifacts
Weaknesses
No statistical inference anywhere
GQ ceiling is confounded by reasoning hints in captions
Headline claim contradicted by own tables
Cross-model comparisons lack denominator control
Internal data inconsistencies and citation sloppiness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'Vision Hurts Rate' in this paper?
It is the fraction of problems a model solves correctly from the text alone that it then fails when the diagram image is added. The paper reports rates up to 46.1% (Qwen2.5-VL 72B on PhysUniBench), showing that for many problems the image input actively degrades rather than helps performance.
Which models were evaluated in this study?
Five models in the main experiments — Gemini-2.5 Flash, Gemma-3 27B, LLaMA-4 Maverick, Qwen2.5-VL 32B and 72B — plus InternS1-mini 9B in a separate case study. No frontier closed models like GPT-4o or Claude were included, which the authors list as a limitation.
Does the paper conclude that perception or reasoning is the bigger bottleneck?
The authors argue perception is the primary bottleneck for most models: accuracy generally rises when human ground-truth captions replace raw images, and some models recover strongly under corrected captions. However, several models show low recovery, indicating a genuine reasoning bottleneck, and for the strongest model (Gemini-2.5 Flash) caption-conditioned results sometimes exceed ground-truth-caption results on geometry.
Is code and data available for this paper?
The paper states that code, prompts, and data are available via the appendix, which contains full inference scripts, caption-generation prompts, and LLM-judge prompts. No external repository link is evident in the extracted text, so independent verification currently relies on the appendix materials.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3 and 4 cannot be inspected from the extracted text; assessment of the perception-accuracy trends relies on captions and surrounding prose.
- Appendix tables appear largely intact, but anomalous values (e.g., 100.5%) could be text-extraction artifacts rather than errors in the original PDF.
- Section cross-references in the extracted text are internally inconsistent (e.g., Appendix F content labeled B.1/B.2, references to a non-existent Section 9), which may partly stem from extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.