SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
Anubhav Khanal, Prabigya Acharya, Roshni Poudel, Sujan Kapali, Bigyan Bhatta, Pramish Paudel, Francois Rameau, Danda Pani Paudel, et al.
cs.CV · cs.AI
Quick verdict
Claims the first hierarchical 3DGS benchmark for VLM spatial reasoning; Gemini-generated ground truth and absent statistics weaken the headline findings.
Summary
SceneBench is a benchmark for evaluating vision-language models' understanding of 3D scenes. It contains 966 photorealistic scenes reconstructed with Gaussian Splatting — 584 from ScanNet++ and 382 from synthetic InteriorGS — organized in a five-level hierarchy (scene, room, functional area, object group, object) with over 183K text-annotated, 3D-bounded nodes produced by a human-in-the-loop pipeline built on Gemini 3.0 (~1,500 human-hours). Three task families are defined: Existence questions (yes/no attributes), Spatial Intelligence questions (counting, size, distance, direction, with ground truth from bounding-box metadata), and multi-step Grounded QRA reasoning across twelve categories, totaling 33,524 questions (7,213 test). Evaluating Gemini-3-flash, Qwen3-VL-8B, agentic methods (CoV, Think3D) and GeoThinker, the authors report up to 85% accuracy on existence tasks but roughly 60% or below on counting and weak multi-step reasoning; finetuning GeoThinker on the data yields the best existence and spatial results, beating 165–354-frame video baselines. Limitations: annotations are VLM-generated with human verification limited to test questions, the quality study covers only 100 questions rated by 7 people without inter-rater statistics, Gemini both generates and is evaluated on the data, no significance testing is reported, and neither dataset nor code is available yet.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SceneBench is the first dataset combining photorealistic 3DGS rendering with densely grounded, hierarchical (scene/room/area/group/object) scene-graph annotations and question answering (Tables 1–2).
- State-of-the-art VLMs drop from up to 85% accuracy on existence tasks to about 60% or below on counting, with weaker multi-step hierarchical reasoning (Table 4).
- Finetuning a geometry-integrated model (GeoThinker) on SceneBench outperforms dense-video VLM baselines on spatial intelligence tasks (Tables 4–5, Fig. 4).
- A ~1,500-human-hour pipeline with iterative verification produces high-quality annotations, evidenced by a perturbed-control user study (Section 4.4).
- CoT prompting improves existence performance but degrades spatial-intelligence performance, consistent with Thinking in Space (Section 5.3).
Strengths
Genuinely novel dataset combination
Tables 1–2 credibly show no prior dataset pairs photorealistic 3DGS rendering with multi-level (S/R/G/O) grounded annotations plus QA; 128,966 objects and 183,540 descriptions averaging 60 words far exceed ScanRefer/ReferIt3D description depth.
Carefully engineered, well-documented annotation pipeline
Section 3 and supplementary Section 8 detail instance-ID-overlaid videos, validation and grounding-correction passes, A* + B-spline camera trajectories, and all generation prompts (Figs 7–14) — unusually reproducible-in-principle documentation.
Principled spatial ground truth
Broad baselines with an actionable finding
Perturbed-sample quality control design
Weaknesses
Ground truth is largely self-generated by the evaluated model
Human audit covers a thin slice
No statistical rigor anywhere
Inconsistent metric application and unexplained anomalies
Availability entirely contingent
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SceneBench (arXiv 2609.16233) is a dataset-and-benchmark paper targeting 3D vision-language understanding. It claims three things: (1) it is the first benchmark to combine photorealistic Gaussian-Splatting scenes with densely grounded, hierarchical scene-graph annotations and question answering (Tables 1–2); (2) current VLMs degrade sharply from basic recognition (up to 85% existence accuracy) to compositional spatial tasks (~60% counting) and multi-step reasoning; and (3) finetuning a geometry-integrated model (GeoThinker) on the benchmark outperforms dense-video VLMs on spatial tasks. The corpus is substantial: 966 scenes (584 ScanNet++, 382 InteriorGS), 128,966 objects, 183,540 descriptions averaging 60 words, and 33,524 questions split over 766 train / 200 test scenes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SceneBench?
SceneBench is a benchmark of 966 photorealistic 3D scenes (reconstructed with Gaussian Splatting) with hierarchical annotations spanning scenes, rooms, functional areas, object groups and objects, plus 33,524 questions testing whether vision-language models can perform spatial reasoning in 3D — object attribute existence, counting, size, distance, direction, and multi-step grounded reasoning.
How well do current AI models perform on SceneBench?
The best models reach about 85% accuracy on simple attribute (existence) questions but fall to roughly 60% or below on counting and struggle on multi-step reasoning. A geometry-aware model (GeoThinker) finetuned on the benchmark data performs best overall, outperforming models given hundreds of video frames.
Is the SceneBench dataset publicly available?
Not yet. The paper states the dataset will be made publicly available upon acceptance; as of this version (arXiv 2609.16233), neither the data nor evaluation code has been released.
How were SceneBench annotations created?
Mostly by the Gemini 3.0 vision-language model watching scene videos with object-ID overlays, followed by iterative human inspection totaling about 1,500 hours. All test-set questions were human-verified; the roughly 26,000 training questions and 183K object descriptions were primarily model-generated with spot checks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 and 15–27 are not analyzable from the extracted text; figure-based results (user-study rating distributions, per-attribute and per-hierarchy-level breakdowns) are taken on trust.
- Extraction noise is present (e.g., the SUNY Stony Brook affiliation renders as 'Korea'; the abstract is truncated mid-sentence), so headline numbers were cross-checked against Tables 4–5 rather than the abstract.
- Supplementary prompts (Figs 7–14) are images; exact prompt wording could not be audited, so annotation-prompt quality judgments rest on the described procedure.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.