Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
Yuyang Dai, Bofei Huang, Hongbo Zhang, Haoran Xie
cs.CV · cs.AI
Quick verdict
Claims VLMs perceive evidence but ignore it; benchmark+SRT prompting show gains require oracle boundary labels — rigorous but small-n, not yet deployable.
Summary
The paper distinguishes perceptual failure (evidence not seen) from process failure (evidence seen but not used to constrain the answer) in vision-language models. It introduces VPAC-Bench, nine real-image process families annotated with activity stages and stage-transition boundaries, and SRT, a State-Relevance-Target prompting family that forces models to bind evidence to decisions. Across five frontier VLMs, default models over-commit on ambiguous referents at 95.8-100% (COCO); an inspect-judge-decide structure cuts over-answer rates below 13% without hurting unambiguous items, though Gemini improves from visual access alone. Transfer is model-sensitive: on VSR only Gemini's +43-point gain clears interval overlap at n=100. On four standardized families, an oracle boundary router lifts physical state transition from 26.7% to 80.0% (five-model average), beating the best CoT baseline by 20.4-35.7 points, while generic SRT never beats CoT. Wrong-boundary and label-only controls rule out prompt specificity and candidate reduction alone. Notably, oracle SRT significantly hurts on Craft and Mobility, which the authors explain via a residual-uncertainty ceiling. Main limitations: the deployable self-router closes almost none of the oracle gap (gains of 0.6-1.7 points, not significant), per-cell samples are small (60-100 items), and the process-content margin is confounded with the structured output schema.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Process failure - committing to answers contradicted by correctly perceived evidence - is distinct from perceptual failure and widespread: default over-answer rates are 95.8-100% on COCO ambiguous referents across five frontier VLMs.
- Only an explicit process structure repairs process failure for four of five models (policy accuracy 87-91%, over-answer below 13%) while identical visual access alone does not; Gemini is the exception, repaired by visual access alone.
- Process-prior transfer is model-sensitive: generic SRT does not reliably outperform strong CoT baselines, and on VSR only Gemini's +43 pp gain clears interval overlap at n=100.
- When the correct decision boundary is known, boundary-aligned SRT beats all five CoT baselines by 20.4-35.7 pp on the four standardized families, and the wrong-boundary control shows alignment, not prompt specificity, drives the gain.
- The marginal value of a process prior is bounded by residual uncertainty: adding an oracle-aligned process description to near-solved tasks (Craft, Mobility) significantly degrades accuracy, an opposite-direction paired effect.
Strengths
Clean causal dissociation of process vs perceptual failure
S1 contrasts marked-no-prior (identical visual annotations, no decision process) with visual sketchpad (same access plus inspect-judge-decide structure): pooled COCO policy accuracy moves 10.7% to 90.4% while Task 2 (unique referents) stays at 99.5-99.6%, jointly ruling out visual-access and always-clarify explanations (Table C2).
Unusually rigorous, self-skeptical statistics
Wilson CIs throughout, McNemar paired tests on all claim-carrying comparisons (p<0.001 in all four standardized families, Table B2), explicit reconciliation of the two accuracy conventions (Table C1), and candid flags that self-router gains (+1.7/+0.6 pp) are not established.
Decisive controls against trivial explanations
Transparent reporting of an inconvenient negative result
Documented annotation quality
Weaknesses
Oracle dependence: no deployable method yet
Underpowered pattern claims
Schema confound in the central mechanism claim
Narrow baseline breadth
Coverage gaps in the benchmark evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims a conceptual and empirical contribution: VLM errors split into perceptual failure (evidence not recognized) and process failure (evidence recognized but not bound to the decision), and the second is widespread yet repairable only by explicit process structure. The artifacts are VPAC-Bench (nine real-image process families, each image annotated with its activity stage and nearby stage transition; Sections 1, 3.3) and SRT, a State-Relevance-Target prompting family (Section 3.2). The three-stage narrative (S1 referent disambiguation, S2 spatial verification, S3 process families) is well constructed: each stage motivates the next, and the claims are stated with unusual precision about what is and is not established. Headline numbers are internally consistent: default over-answer rates of 95.8-100% on COCO (Table 2), policy accuracy 87-91% under the inspect-judge-decide condition for four of five models, and oracle boundary-router gains of +53.3 pp on physical state transition (26.7% to 80.0%, five-model averages, Table C4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'process failure' in vision-language models?
It is the failure mode where a model correctly perceives the relevant visual evidence (e.g., it lists several cars of different colors) but still commits to a single confident answer that its own observations do not support. The paper distinguishes this from perceptual failure, where the evidence is not seen at all, and measures it as process-answer inconsistency across five frontier VLMs.
What is VPAC-Bench?
VPAC-Bench is a benchmark of nine families of real-image process tasks (sports action phase, mobility, tool use and repair, household activities, drawing and craft, assembly, physical state transition, navigation/traffic, and object-use affordance). Each image is annotated with its current activity stage and the nearby stage transition it tests, with human-reviewed labels (30 annotators, Cohen's kappa 0.72 on a double-annotated subset).
What is SRT prompting and does it improve VLM accuracy?
SRT (State-Relevance-Target) is a structured prompting family that forces the model to extract decision-relevant evidence, verify only the relations that matter, then commit only to evidence-supported answers. Its value is conditional: generic SRT often fails to beat chain-of-thought baselines, and the large gains (e.g., physical state transition 26.7% to 80.0%) appear only when the correct decision boundary is supplied - which in this paper requires oracle ground-truth labels. A model-based self-router currently closes almost none of that gap.
Can this method be deployed in practice today?
Not yet, by the authors' own assessment. The boundary-aligned intervention that produces the large gains depends on knowing the correct decision boundary for each image in advance. The automatic self-router predicts boundaries at only 35.0-61.7% accuracy and its gains over generic SRT (+0.6 to +1.7 points) are not statistically significant, so the paper positions boundary-aligned SRT as a diagnostic upper bound rather than a deployable system.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are described in captions only; visual content could not be verified from extracted text.
- Table B1's supporting-family rows are partially garbled in extraction (e.g., truncated Tool-use row); supporting-family paired numbers were cross-checked against Tables B1'/B2' where possible.
- The paper states prompts are 'reported in full in the released code' but no repository URL appears in the extracted text; code/data availability should be verified before acceptance.
- Results are tied to specific model versions (GPT-5.4-mini, Gemini-3.1-Pro, GLM-4.6V, Mistral-Small-3.2) and will age quickly; absolute numbers are version-specific.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.