GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures
cs.CV · cs.LG · cs.RO
Quick verdict
Factorized benchmark separating identity vs region effects in VLM affordance prompts; rigorous and unusually honest, but only 19–28 objects per stratum.
Summary
GroundBench is a diagnostic benchmark that decomposes why naming a target part in VLM manipulation prompts boosts action accuracy. Building on an unnamed companion evaluation that reported 0.32–0.63 gains across eight models, it uses six branch-and-merge conditions adding one controlled information bundle per edge (image only, object named, part named, localization only, localization shown, mechanics shown), plus a counterfactual re-ask naming a real noncanonical part from the same image. On 19 curated GAPartNet objects, three OpenAI models produced 1,068 predictions: supplying the region without identity left accuracy at 0.26, 0.26 and 0.53 — at or below the 0.53 majority baseline despite near-exact reproduction of the supplied box — while supplying identity yielded 0.74, 0.68 and 0.68. A computed leakage check shows every above-baseline gain occurs where the part category alone determines the action; a no-vision control leaves GPT-5 unchanged or improved, while GPT-4o mini drops on one condition. Ground-truth mechanics never help (five drops, one tie across six comparisons). On 74 counterfactual pairs GPT-5 reaches 0.86 pair-weighted compliance but fails all nine push-to-lift-vertical transitions. Main limitations: tiny audited sets (19/28 objects, 74 pairs), single-sample decoding, no counterfactual no-vision control, and 12 of 19 boxes only approximate.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Supplying the benchmark's target region without its identity leaves all three OpenAI models at or below the 0.53 majority-action baseline (0.26, 0.26, 0.53) on 19 curated GAPartNet objects despite near-exact reproduction of the supplied region.
- Supplying target identity without location yields 0.74, 0.68 and 0.68, and every above-baseline gain occurs under a condition where the supplied part category alone determines the action in the curated set.
- A no-vision control leaves GPT-5 unchanged or improved on all leak-flagged conditions, evidence consistent with category-to-action association contributing substantially to the companion evaluation's 0.32–0.63 named-part gains.
- Adding ground-truth joint type and motion axis never improves action accuracy (five drops, one tie across six model-stratum comparisons).
- GPT-5 follows the counterfactual part request on 0.86 of 74 pairs at a 0.07 shortcut rate but fails all nine push-to-lift-vertical transitions.
Strengths
Genuinely factorized condition design
Branch-and-merge adds one information bundle per edge; localization-only vs part-named (0.26/0.26/0.53 vs 0.74/0.68/0.68 in Table 1) cleanly separates region from identity — the exact contrast the companion evaluation could not run.
Data-derived safeguards against trivial explanations
Localization thresholds derive from target-center spread; constant image-center (0.21 hit rate) and object-center baselines appear alongside every figure; a computed leakage column in Table 1 exactly predicts the upper/lower block split, preventing a repeat of the companion evaluation's 0.929 constant-baseline error.
No-vision control run at scale
Transition table beats a pooled rate
Exceptional error transparency
Weaknesses
Tiny audited evaluation sets
Single sample per prompt, no hypothesis tests
Design cannot positively demonstrate visual-mechanical reasoning
Curated UMD stratum is uninformative for action choice
Missing rather than run controls
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
GroundBench (arXiv 2609.13308) — Structured Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of the GroundBench paper?
That giving a vision-language model the exact image region of a target part, without naming the part, does not improve action accuracy: three OpenAI models scored 0.26–0.53 on 19 articulated objects against a 0.53 constant-answer baseline, while naming the part scored 0.68–0.74. Because every gain occurs where the part's category name alone determines the action, the gains may reflect text association rather than visual reasoning.
What is a counterfactual re-ask in GroundBench?
The benchmark re-issues the prompt on the same image but names a real, non-canonical part (for example, a handle instead of a hammer head) as the target. A model answering with the original part's action is scored as a shortcut. GPT-5 complied on 0.86 of 74 such pairs but failed all nine requests that required switching from push to lift-vertical.
Which models were evaluated and how much did it cost?
GPT-4o mini, GPT-5 mini and GPT-5 produced 1,068 scored predictions for $4.17, plus 342 text-only control predictions for $1.06. A separate Qwen3-VL-8B open-weight pilot covered 50 frame-level examples from the bulk UMD stratum.
Is the GroundBench dataset and code available?
Yes. The benchmark, dataset converters, evaluator and the exact build commands behind every reported count are released on the paper's GitHub repository, with deterministic conversions auditable against source annotations and checksummed manifests for every rendered image.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.