IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models
Yiqi Wang, Zhifeng Rao, Jiaqi Zhang, Jiaqi Zhang, Xiaoyang Li, Zhangkai Wu, Yiqun Duan, Mingkai Zheng, et al.
cs.RO · cs.AI
Quick verdict
Six-model VLA/WAM benchmark shows clean LIBERO near-saturated (1.6 pp) while robustness/paraphrase separate 15–31 pp; rigorous evidence tiers, but n=3 runs.
Summary
IndustrialVLA-Bench evaluates six publicly released robot policy models — four vision-language-action models (π0.5/OpenPI, UnifoLM-VLA-0, Xiaomi-Robotics-0, GR00T-N1.7) and two world-action models (FastWAM, Cosmos Policy) — under a unified, traceable protocol across four axes: clean capability on LIBERO, non-language robustness on LIBERO-Plus, instruction paraphrase sensitivity on LIBERO-Para, and observed deployment cost (latency, peak VRAM, runtime mode). Each configuration aggregates three complete runs with fixed seeds {1,7,42}; every row carries an evidence status (protocol-faithful, near-reproduction, pending verification). The central empirical finding is that clean LIBERO is near-saturated — all six models fall within 1.58 points (96.52–98.10%) — while robustness spans 14.62 and paraphrase success 31.08 points; restricting to the three protocol-faithful systems preserves the effect (1.36/14.62/23.10). Clean rank correlates negatively with robustness (Spearman -0.37) and visual versus language robustness are nearly uncorrelated (0.09), so the axes are not redundant. Camera and robot perturbations are the dominant failure modes (cross-model averages 58.26% and 60.35%). FastWAM shows the largest paraphrase collapse (51.16%, -45.39 points). Limitations: only three of six rows are protocol-faithful; n=3 descriptive deviations without confidence intervals; a purposive, non-exhaustive model set; simulation-only coverage; and latency/memory not hardware-normalized. Code, manifests, and per-run logs are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Clean LIBERO is near-saturated for released policies, with all six evaluated models within a 1.58-point range (96.52–98.10%, Table 2), so clean scores cannot discriminate systems.
- Non-language robustness (14.62-point span) and paraphrase sensitivity (31.08-point span) provide far greater diagnostic separation, and the effect persists in the protocol-faithful subset (1.36/14.62/23.10 points).
- Clean rank does not predict robustness (Spearman -0.37), and visual versus language robustness are nearly uncorrelated (0.09), so the diagnostic axes must be reported separately.
- Camera viewpoint and robot appearance/embodiment shifts are the dominant failure modes, with cross-model success of only 58.26% and 60.35% respectively (Table 3).
- Comparability must be earned: only protocol-faithful rows (OpenPI, GR00T-N1.7, FastWAM) support strict rankings, while NR/PV rows are retained as diagnostic coverage only.
Strengths
Evidence-tier governance attached to every result
PF/NR/PV status is enforced, not decorative: Xiaomi-Robotics-0's clean-leading 98.10% (Table 2) is excluded from rank claims, and Appendix B/C define comparability gates, command manifests, and a promotion checklist that make the evaluation path itself an auditable artifact.
Demonstrated non-redundancy of evaluation axes
§4.7 quantifies axis separation (clean vs robust Spearman -0.37; robust vs paraphrase 0.09) instead of asserting it, and §4.4 shows the two language protocols disagree (rho=0.60, 13.47-point difficulty offset), justifying separate tracks.
Scale and protocol consistency unusual for policy evaluation
Genuinely informative empirical findings
Candid limitations and anti-overclaim framing
Weaknesses
Strict-comparison core is only three models
Statistical treatment is descriptive only
Deployability axis is not a controlled comparison
Capability axis is near-information-free
Failure taxonomy has no empirical backing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
IndustrialVLA-Bench (arXiv 2609.25562) is a benchmark/evaluation-protocol paper that independently executes six publicly released robot-policy checkpoints — four VLAs (π0.5/OpenPI, UnifoLM-VLA-0, Xiaomi-Robotics-0, GR00T-N1.7) and two world-action models (FastWAM, Cosmos Policy) — on three LIBERO-family tracks (clean LIBERO, LIBERO-Plus perturbations, LIBERO-Para paraphrases) plus observed deployment metadata. Its claims are deliberately scoped: (1) clean LIBERO is near-saturated, with a 1.58-point spread (96.52–98.10%, Table 2); (2) robustness (14.62 points) and paraphrase sensitivity (31.08 points) discriminate far more, and the effect survives restriction to the three protocol-faithful (PF) systems (1.36/14.62/23.10); (3) the axes are not redundant (Spearman clean-vs-robust −0.37, robust-vs-para 0.09, §4.7); (4) only PF rows support strict ranking; (5) no causal attribution to VLA-vs-WAM architecture is attempted. "Industrial" denotes commercially released systems, not an industrial task domain — a clarification the authors correctly foreground in §1.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IndustrialVLA-Bench?
It is an evaluation protocol and benchmark that runs six publicly released robot policy models (four vision-language-action models and two world-action models) through the same LIBERO-family evaluation — clean tasks, visual/embodiment perturbations, and instruction paraphrases — plus latency and memory measurements, tagging every result with an evidence status: protocol-faithful, near-reproduction, or pending verification.
Which model performed best?
No model dominated every axis. Clean LIBERO scores were nearly tied (96.52–98.10%). OpenPI had the best robustness average (85.31%), UnifoLM-VLA-0 the best paraphrase success (82.24%, though at unverified evidence status), GR00T-N1.7 the lowest latency (19.13 ms/call) and memory (7,810 MiB), and FastWAM the worst paraphrase robustness (51.16%).
Is the code and data available?
Yes. The paper links a GitHub repository (xiaoqi-7/IndustrialVLA-Bench) containing the evaluation harness, command manifests, normalized result records, and per-run logs. The evaluated model checkpoints remain under their original licenses and are not redistributed.
What is the practical takeaway for choosing a robot policy model?
Clean benchmark success is near-saturated and does not predict robustness. Before choosing, check perturbation-specific robustness, paraphrase sensitivity, and deployment cost per model. Camera viewpoint changes and robot appearance changes are the most damaging perturbation types, with cross-model success averaging only 58.26% and 60.35% in this study.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 could only be checked against embedded axis/tick text; the visual content was not inspectable.
- The abstract in the supplied metadata is truncated mid-sentence, but the full ABSTRACT section is present in the extracted paper text.
- The linked GitHub repository contents were not inspected (offline review); artifact claims rest on the paper's description.
- The supplied metadata lists 8 authors while the extracted title block lists 10 (adding Shan You and Taotao Cai); the affiliation-to-author mapping may be an extraction artifact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.