New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models
Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar, Shaswati Saha, Yuxuan Jiang, Siyuan Li, Qiheng Wang
cs.CV · cs.AI · cs.CL · cs.LG
Quick verdict
Paired benchmark tests whether VLMs change experiment choice when evidence changes; rigorous design but six open models and exploratory status limit scope.
Summary
This paper introduces a controlled benchmark testing whether vision-language models (VLMs) can decide when a physical question is already answerable and, when it is not, which additional experiment to perform. Each of 144 parameter families across sliding, bouncing, and spring systems shows one measurement image and four candidate worlds (two masses × two values of friction, restitution, or stiffness); the model must stop and answer or pick the cheapest experiment that resolves the question from a priced menu. Matched pairs change only the image or the threshold, guaranteeing the correct action changes. Across six open models (Qwen2.5-VL 3B/7B/32B, SmolVLM2 2.2B, Idefics3 8B, Pixtral 12B), direct responses repeat the same action for 95.1–100% of image pairs; brief reasoning raises switching, but the best pair score is 5.9% (Qwen 7B). Decomposition shows Qwen 3B re-measures the known property on 83.5% of property purchases, several models answer UNDETERMINED almost universally after tests, ruler reading drops to 2.8% (Idefics3), and answer accuracy with supplied sufficient evidence peaks at 34.6%. Prompt variants have mixed effects (text readings add 9.0–14.2 points for four models but cost Pixtral 35.8). Limitations include open models only, at most one extra experiment, idealized deterministic systems, exploratory status after planned accuracy gates failed, acknowledged parser misreads of some reasoning outputs, and missing Qwen 32B brief-reasoning conditions.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Direct responses repeat the same action for 95.1–100% of image pairs even though the correct action changes, across six open models (Section 5.1, Table 13).
- Brief reasoning increases action switching, but the best model (Qwen 7B) answers both members of an image pair correctly for only 5.9% of pairs (Table 12).
- Choosing a useful experiment and correctly interpreting its result are separable failure modes (Tables 3, 17; Appendix D).
- Average final-answer scores conceal fixed decision rules, so conventional physical-reasoning benchmarks cannot measure evidence-adaptive experiment selection (Section 1).
Strengths
Paired construction yields an exact, computable reference action
Four equally likely worlds per family plus priced experiment menus make the optimal choice derivable in closed form (Eq. 3–4, Section 3.3); Table 9 documents render-level validation (288/288 text checks, 1,152/1,152 identical-byte image checks, max marked-position extraction error 0.456% of panel span).
Indistinguishable-alternative control rules out lucky-guess inflation
Appendix D: across 576 pairs where no extra measurement can split the remaining worlds, all models give identical decoded responses and definite answers score exactly 50% — evidence the benchmark cannot be gamed by guessing.
Unusual transparency about protocol deviations and exploratory status
Failure decomposition separates capabilities the field usually conflates
Reproducibility scaffolding is unusually complete
Weaknesses
External validity capped by model pool
Reasoning-protocol contrast is confounded
Exploratory analysis path with no multiplicity control
Narrow task envelope
Incomplete coverage for the largest model
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.11022) targets a decision that standard physical-reasoning benchmarks never measure: given one measurement image and a question about a new trial, should a vision-language model (VLM) answer now, and if not, which additional experiment is worth its price? The authors build a setting where ground truth is analytic. Each of 144 parameter families (48 each for sliding, bouncing, spring) fixes two possible masses and two values of friction μ, restitution e, or stiffness k, giving four equally likely worlds (Section 3.1, Eq. 1). One rendered measurement (coasting distance, rebound height, load-extension) eliminates two worlds. Thresholds are placed so that either changing the image or changing the threshold flips whether the answer is determined (Section 3.2, Eq. 2), and the optimal action is the cheapest resolving experiment (Eq. 4). Central claims: (1) direct-response VLMs repeat the same action on 95.1–100% of image pairs whose correct actions differ; (2) brief reasoning helps, but the best model (Qwen 7B) solves both members of only 5.9% of pairs; (3) evidence selection and answer production fail separately.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'New Evidence, Same Choice' show about vision-language models?
It shows that six open VLMs almost never change their experiment-selection decision when the presented measurement or the question changes, even though the correct action changes by design: direct answers repeat the same action on 95.1–100% of matched image pairs, and with brief reasoning the best model (Qwen 7B) gets both sides right on only 5.9% of pairs.
Which models were evaluated in this benchmark?
Qwen2.5-VL at 3B, 7B, and 32B, SmolVLM2 2.2B, Idefics3 8B, and Pixtral 12B — all open-weight, run unquantized with greedy decoding. No closed or frontier models were tested.
How is this different from physical-reasoning benchmarks like PhysBench or Physion?
Those score final answers. This benchmark scores the decision of whether to answer now or which additional experiment to buy, with all possible worlds, experiment costs, and the optimal action known exactly; matched problem pairs ensure a fixed decision rule earns no credit.
Is code or data available for the physical probe benchmark?
The paper links a project page (sourajitcs.github.io/physicalprobe) but the text states no explicit code or dataset release; seeds, sampling ranges, and calibration constants are published in the appendix to support reimplementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels are not extractable from the PDF text; visual judgments rely on captions, prose, and tables rather than the images themselves.
- Some table layouts (e.g., Table 4's qualitative panels) are reflowed in extraction; cell-level readings of those examples were cross-checked against the surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.