Part Grounding, Not Action Knowledge: Locating the Bottleneck in VLM Affordance Prediction
cs.CV · cs.LG · cs.RO
Quick verdict
Claims VLM affordance failure is part grounding, not action knowledge; effect large and consistent but n=19, gold-name upper bound only, no statistics.
Summary
This single-author empirical study asks why vision-language models fail at affordance prediction on articulated objects: do they lack action knowledge, or attend to the wrong part? Using 19 articulated objects from GAPartNet/PartNet-Mobility (labels mapped deterministically from part category: button to push, drawer/door to pull, lid to lift-vertical; distribution 10/8/1), the author queries eight VLMs from three developers (Qwen3-VL-8B, three OpenAI, four Anthropic models) under three prompts across 456 calls. When models choose the part themselves (prompt A), push appeared once in 64 evaluations where it was correct, accuracy spanned 0.158-0.474, and no model beat a constant 'pull' baseline (0.526). Naming the target part (prompt C) raised accuracy by 0.32-0.63 to 0.684-0.947, restored push recall to 7-8/8, and moved all eight models above the constant; free prose without labels (prompt B) elicited pressing language on 6-8 of 8 push objects, showing the vocabulary existed. Spatial localization points the same way: on renders no model beats an image-ignoring constant. The author concludes part grounding, not missing action knowledge, is the dominant bottleneck. Notable honesty includes two self-reported measurement errors (a lenient spatial threshold and a labelling rule wrong on 4/19 objects). Main limitations: only 19 objects with single runs per cell, no statistical testing, gold part names merely bound what a real part detector would provide, and the five-label set and prompts are not fully specified.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 16 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under an open part-selection prompt, VLMs almost never produce 'push' when correct (once in 64 evaluations) and no model beats a constant 'pull' answer (0.526).
- Naming the target part raises action accuracy by 0.32-0.63 for every model (to 0.684-0.947) and restores push recall to 7-8/8, showing the action vocabulary was always available.
- Part grounding, not missing action knowledge, is the dominant bottleneck, consistently across all three model families and not diminishing with capability.
- Condition C supplies the grounding variable, so results bound what a perfect part detector would offer rather than demonstrating a general model of mechanics.
- Spatial localization mirrors the pattern: on rendered objects no model beats an image-ignoring constant, and on photographs only three of eight clearly do.
Strengths
Clean causal decomposition of a conflated failure mode
The three-prompt design isolates the two bundled inferences: A vs C (Section 2) separates part selection from action mapping, and B rules out label-set suppression — free prose still yields pressing language on 6-8/8 push objects (Section 3.2).
Exemplary baseline discipline and self-audit
The constant-pull baseline (0.526) is run against every condition, and Section 3.4 documents two of the authors' own measurement errors — a hand-set 0.15 threshold letting a constant score 0.929/0.947 hit rate, and a joint-geometry labelling rule wrong on 4/19 objects — both caught only by testing against trivial alternatives.
Effect consistency across independent model families
Full audit trail for labels
Honest scoping of the central claim
Weaknesses
Very small and skewed stimulus set
No statistical inference or repeat runs
Gold-name grounding bounds the pipeline claim
Underspecified evaluation artifacts
Thin related work and a minor internal inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This single-author study (arXiv 2609.13225) decomposes the affordance-prediction failure of vision-language models on articulated objects into two bundled inferences: finding the relevant part (grounding) and mapping that part to an action. The central claims are sharply quantified: under prompt A (model chooses the part), push appears once in 64 correct-case evaluations, accuracies span 0.158-0.474, and no model beats a constant pull answer at 0.526 (Table 1); naming the part (prompt C) raises accuracy 0.32-0.63 to 0.684-0.947 with push recall of 7-8/8; free prose without labels (prompt B) still elicits pressing language on 6-8 of 8 push objects. The conclusion — part grounding, not missing action knowledge, is the dominant bottleneck — is carefully bounded in Section 4: the labels reduce to part-category-to-conventional-action mapping, and C merely bounds what a perfect part detector would offer.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Across eight vision-language models, models almost never answer 'push' when they must choose which object part to act on (once in 64 correct-case evaluations), and none beats always answering 'pull'; once the part is named in the prompt, accuracy jumps from 0.158-0.474 to 0.684-0.947. The authors conclude the bottleneck is finding the right part (grounding), not knowing what action it requires.
What data and models were used in the study?
19 articulated objects from GAPartNet/PartNet-Mobility rendered in closed pose, plus 28 real tool photographs from the UMD Part Affordance dataset for a spatial task. Models: Qwen3-VL-8B, gpt-4o-mini, gpt-5-mini, gpt-5, Claude Haiku 4.5, Sonnet 5, Opus 5, and Fable 5 — 456 evaluation calls in total.
Does this paper prove that VLMs have physical or action knowledge?
No. The action labels reduce to mapping a part category (button, drawer, door, lid) to its conventional action, and naming the part hands the model the grounding variable. The result bounds what a perfect part detector would add; it does not demonstrate general mechanical reasoning.
Is code or data available for this paper?
Yes. The paper links a GitHub repository (vlm-physical-affordances) and states the machine-readable action-label audit table ships with the code. The underlying datasets (GAPartNet, UMD Part Affordance) are public, though seven of the eight evaluated models are proprietary APIs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Appendix audit table (Table 3) rows are interleaved with reference entries [3]-[5] by PDF extraction; all 19 rows remain legible and were used accordingly.
- Figures, if any exist in the published PDF beyond tables, are not represented in the extracted text; the review covers only textual content and Tables 1-3.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.