Towards Generalizable Visually Grounded Exploration of Household Devices
Linhao Zheng, Zeming Liu, Wangke Chen, Li Zeng, Wanxiang Che, Heyan Huang, Yuhang Guo
cs.AI
Quick verdict
Solid new manual-free device-exploration benchmark with rich diagnostics; single-run evals, synthetic renders, and a 3x Gemini outlier temper its conclusions.
Summary
VGEBench is a new interactive benchmark evaluating whether vision-language models can operate unfamiliar 3D household devices without manuals or task-specific training. The authors collect 968 device models across 26 categories from Sketchfab and 3D Warehouse, render 7,888 views, annotate 7,264 interactive-component bounding boxes, and instantiate 968 per-device state machines from 26 category-level templates, yielding 14,953 instruction-following episodes. A deterministic simulator validates each action's coordinates, action type, and parameters against the state machine and returns outcome-dependent feedback, enabling multi-turn Hypothesis-Interaction-Refinement loops under global and local interaction budgets. Six VLMs evaluated with ReAct show large gaps: Gemini-3-Flash reaches 54.27% success rate, the runner-up Doubao-1.5-Thinking-Vision-Pro attains 16.68%, and all others stay below 15%. Even the best model achieves only 12.94% perfect sub-task success, indicating success comes mainly from feedback-driven exploration rather than zero-shot knowledge. Error analysis of the best model's failures attributes 66.1% to fine-grained visual grounding rather than navigation, and a human baseline (74.50% SR on 149 episodes) confirms a substantial remaining gap. Robustness checks cover instruction generators, styles, history length, and visual perturbations. Limitations include single-run evaluation without significance testing, synthetic orthographic renders with limited realism, and dependence on closed-source model APIs.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing benchmarks cannot measure manual-free, feedback-driven device exploration because they rely on annotated trajectories or explicit documentation (Table 1).
- VGEBench operationalizes this capability via a Logic-Driven State Machine with deterministic multi-turn feedback over 14,953 episodes on 968 devices in 26 categories.
- Current VLMs largely fail the task: only Gemini-3-Flash exceeds 50% SR (54.27%); the runner-up reaches 16.68% and all other tested models stay below 15% (Table 3).
- Success comes predominantly from feedback-driven exploration rather than zero-shot knowledge, as shown by the PSR/SSR gap (12.94% vs 62.86% for the best model, Table 4).
- Fine-grained visual grounding — not navigation — is the primary bottleneck (66.1% of the best model's errors, Appendix B; Table 5).
Strengths
Deterministic state-machine evaluation design
Sections 3.4 and D: every action is validated against the SSM state on coordinates (BBox membership), action type and parameters, with three-level outcome feedback; state-based (not trajectory) completion and hidden component names eliminate LLM-judge noise and textual shortcuts.
Diagnostically decomposed metric suite
Nine metrics (Section 4.2, Appendix F) separate navigation (VSPS/GVPE), grounding (EIR/TIR) and exploration (EER); RQ3 pinpoints grounding as the bottleneck — GPT-5-mini's weighted coordinate distance is 77.3% worse than Gemini's (290.33 vs 163.75, Table 5), and 66.1% of errors are grounding vs 16.7% navigation (Appendix B).
PSR-vs-SSR analysis substantiates the core framing
Unusually thorough robustness battery
Scale with documented QC and contamination honesty
Weaknesses
No statistical rigor around model comparisons
Unexplained 3.3x Gemini outlier
Missing decomposition conditions and scaffold ablation
Synthetic domain and partially circular quality validation
Floor effects and thin human-baseline detail
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a benchmark paper in embodied AI / multimodal evaluation. The authors argue that existing settings cannot measure *generalizable visually grounded exploration* — operating novel household devices without manuals — because prior work either trains on environment-specific trajectories (ALFWorld/BEHAVIOR-style IL/RL) or assumes explicit documentation (tool-use, GUI agents; Table 1). Three claims are made: (1) VGEBench fills this gap via a Logic-Driven State Machine with deterministic multi-turn feedback (Sections 3.2–3.4); (2) current VLMs largely fail — only Gemini-3-Flash exceeds 50% SR (54.27%), runner-up Doubao-1.5-Thinking-Vision-Pro reaches 16.68%, all others below 15% (Table 3); (3) the dominant bottleneck is fine-grained visual grounding, not navigation (66.1% of the best model's errors, Appendix B). Within the benchmark's own terms the claims are mostly supported; the caveats below concern external validity and statistical strength.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VGEBench?
VGEBench is an interactive benchmark (arXiv 2609.00845) that tests whether vision-language models can figure out how to operate unfamiliar 3D household devices — such as coffee machines, washing machines and lamps — purely through visual exploration and multi-turn feedback, without manuals or task-specific training. It contains 968 device models across 26 categories and 14,953 instruction-following episodes driven by deterministic device state machines.
How well do current models perform on VGEBench?
In the paper's evaluation, Gemini-3-Flash is the only model above 50% task success (54.27% SR); the runner-up, Doubao-1.5-Thinking-Vision-Pro, reaches 16.68% and all other tested VLMs stay below 15%. A human baseline reaches 74.50% success on a 149-episode subset, so a large human-machine gap remains.
What is the main bottleneck the paper identifies?
Fine-grained visual grounding: precisely localizing and acting on small interactive components (buttons, knobs, ports). About 66.1% of the best model's errors are grounding failures (invalid coordinates or wrong component selection), far exceeding navigation errors (16.7%) or action-logic errors (17.2%).
Is the VGEBench dataset and code available?
Yes. The paper states the dataset and code are publicly available at github.com/BITHLP/VGEBench. Note that re-running closed-source model results requires paid API access, while results for open-source models (e.g., Qwen3-VL-8B, InternVL3.5-8B) can in principle be reproduced locally on GPU hardware.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure interiors (Figs. 1-6) cannot be inspected from the text extraction; only captions and scattered axis-label fragments are present, so figure-level judgments rely on the surrounding prose and tables.
- Table 1's support markers extracted as placeholder glyphs; per-benchmark support ratings of competitor benchmarks were therefore not relied upon in this review.
- The extraction is complete through Appendix I (including Listings 1-2); no truncation was detected at the end of the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.