RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?
Zhenxuan Fan, Bo Zhang, Yutong Lin, Yuqian Yuan, Juekai Lin, Liang Liang, Zhuoyi Huang, Wenqiao Zhang, et al.
cs.RO · cs.AI · cs.CV
Quick verdict
Claims a 527K-trajectory, 280-variant benchmark diagnosing VLA spatial/planning failures; rigorous and diagnostic, but experiments use only a slice of it.
Summary
The paper introduces RoboSPA, a simulated benchmark and dataset built on SAPIEN and the RoboTwin 2.0 framework for diagnosing Vision-Language-Action (VLA) models along two axes: fine-grained spatial reasoning (five categories, e.g., canonical position indexing, cross-view reasoning) and long-horizon procedural planning (five categories, e.g., order-constrained execution, memory-intensive planning). Fifty-six base tasks, each at five difficulty levels (more candidate objects or longer action sequences), yield 280 variants, 527K scripted trajectories, 997 hours of video, and five embodiments. Beyond success rate, it proposes ONTA, a chance-corrected target-accuracy metric, and a subtask Progress Score. Four fine-tuned baselines (RDT, GO-1, π0.5, X-VLA), trained per task on 250 trajectories of clean Aloha-AgileX data, degrade sharply with difficulty: π0.5 falls from 55.2% (L1) to 22.3% (L5); RDT and GO-1 show negative ONTA, i.e., below-chance target selection; and all models score 0% at L5 on memory-intensive planning. Failure analysis attributes over 70% of spatial errors to target grounding and about 50% of planning errors to low-level manipulation. Main limitations: experiments exercise only a small fraction of the dataset, evaluation is simulation-only, and baselines exclude memory-augmented or zero-shot alternatives.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
6 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RoboSPA is, to the authors' knowledge, the first VLA dataset to make fine-grained spatial reasoning a core evaluation dimension, paired with long-horizon procedural planning across 56 base tasks and 280 difficulty variants.
- Current fine-tuned VLA models degrade sharply with task complexity: the best model, π0.5, falls from 55.2% overall success at L1 to 22.3% at L5, and all four evaluated models score 0% on the hardest Memory-Intensive Planning level.
- Chance-corrected ONTA shows RDT and GO-1 mostly select targets at or below chance, and failure analysis attributes over 70% of spatial errors to target grounding versus only a small share to low-level execution.
- Progress Score reveals substantial partial task completion masked by binary success rates, with planning failures driven by accumulated errors, wrong ordering, and memory loss rather than total inability.
Strengths
Controlled difficulty ladder is the core design win
Difficulty is operationalized per axis — candidate-object count for spatial tasks, action-sequence length for planning (§3.2.2, Fig. 2b) — producing clean degradation curves across L1-L5 (Table 2, Fig. 4) instead of ad-hoc task mixes.
Diagnostic metrics with internally consistent math
ONTA (Eq. 1) chance-corrects via the 1/n baseline and Progress Score (Eq. 2) captures partial completion; the formula checks out against reported values (e.g., RDT GAC L1 SR 10.0% at n=2 → ONTA -80.0, Table 3), and Tables 3-4 expose near-chance grounding and hidden partial competence that binary SR masks.
Control experiments isolate the reasoning layer
Failure typology is genuinely diagnostic
Above-average transparency for a benchmark paper
Multi-embodiment and scene-robustness infrastructure
Weaknesses
Experiments exercise only a small fraction of the advertised dataset
ONTA conflates grounding with execution
Baseline coverage is thin relative to the diagnosis
Internal inconsistencies and uneven statistical reporting
Scale and novelty framing slightly overstated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RoboSPA (arXiv 2609.05324) is a dataset-and-benchmark paper for vision-language-action (VLA) manipulation. The authors make three claims: (1) a controlled two-axis taxonomy — Fine-Grained Spatial Reasoning (GAC, SDE, CPI, RRR, CVR) and Long-Horizon Procedural Planning (RPF, OFE, OCE, CAC, MIP) — instantiated as 56 base tasks × 5 difficulty levels = 280 variants; (2) a large corpus of 527K scripted trajectories, 997 hours, 108M timesteps, five embodiments (Aloha-AgileX, ARX-X5, Piper, Franka, UR5), and clean plus domain-randomized scenes (64K/463K); (3) the finding that fine-tuned VLA models collapse under complexity — π0.5 falls from 55.2% (L1) to 22.3% (L5) overall success (Table 2), RDT and GO-1 select targets below chance (negative ONTA, Table 3), and all four models score 0% at L5 on Memory-Intensive Planning. The claim of being 'the first VLA dataset to make fine-grained spatial reasoning a core evaluation dimension' (§1) is defensible but should be read against LIBERO-Pro/LIBERO-plus and Colosseum, which probe spatial robustness more narrowly.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RoboSPA?
RoboSPA (Robot Spatial-Procedural Assessment) is a large-scale simulated benchmark and dataset for robotic manipulation, built on the SAPIEN simulator and RoboTwin 2.0 framework. It evaluates Vision-Language-Action (VLA) models along two axes — fine-grained spatial reasoning and long-horizon procedural planning — via 56 base tasks, each at 5 difficulty levels (280 variants), with 527K trajectories, 997 hours of video, and 5 robot embodiments.
How well do current VLA models perform on RoboSPA?
Poorly at high difficulty. Four fine-tuned models (RDT, GO-1, π0.5, X-VLA) all decline from level 1 to level 5: the best, π0.5, drops from 55.2% to 22.3% success; RDT and GO-1 select manipulation targets at or below random-chance levels on spatial tasks; and every model scores 0% success on the hardest memory-intensive planning tasks.
Is RoboSPA simulation-only or does it work on real robots?
It is simulation-only (SAPIEN/RoboTwin 2.0), with clean and domain-randomized tabletop scenes across five simulated robot arms. The authors explicitly acknowledge that the sim-to-real gap may limit direct transfer to physical robots; no real-robot experiments are reported.
What are the ONTA and Progress Score metrics in RoboSPA?
ONTA (Object-Normalized Target Accuracy) is a chance-corrected success measure for spatial tasks: it rescales success rate by the 1/n random-guessing baseline of picking among n objects, so 0 equals random selection and negative values mean below-chance. Progress Score measures partial completion of long-horizon tasks, averaging the fraction of subtasks completed per episode on a 0-100 scale.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded text (Figs. 3-6) is garbled and duplicated in extraction; figure-internal values (e.g., Fig. 3 panel splits, Fig. 4 curves) were read with reduced reliability, while all tables (2-24) extracted cleanly and form the basis of the numeric claims here.
- Figures 5, 6, 9-20 are present only as captions, labels, and embedded instruction text in the extraction; their visual content (radar plots, rollout frames) could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.