VLAQuantBench: Closed-Loop Evaluation of Post-Training Quantization for Vision-Language-Action Models
Jiuyi Xu, Qing Jin, Meida Chen, Song Wang, Yang Sui, Yangming Shi
cs.RO · cs.AI
Quick verdict
Benchmark with 409 runs and 94,574 episodes shows VLA quantization effects are recipe-dependent; a rigorous, transparent evaluation with actionable findings.
Summary
The paper introduces VLAQuantBench, a controlled benchmark for post-training quantization (PTQ) of vision-language-action (VLA) models, comprising 409 evaluation runs and 94,574 simulation episodes across four released policies (OpenVLA-OFT, π0, π0.5, X-VLA) on LIBERO, with X-VLA additionally on SIMPLER, CALVIN, and VLABench. Using a shared round-to-nearest quantizer across six numerical formats (W3-W8A8), the authors isolate the effects of layer scope, numerical format, and calibration on closed-loop task success. Key findings: (1) quantization effects are non-monotone and non-additive — expanding π0.5's quantized action-head subset from 126 to 167 layers raises success from 7.0% to 70.5% under uncalibrated W4A4, while isolated attention/adaRMS losses (-1.0/-5.5 pp) fail to predict the joint -97.5 pp; same-observation replay over 219 chunks confirms the numerical interaction precedes environment feedback. (2) Two-episode SmoothQuant-style calibration removes severe joint failures in π0.5 but lowers π0 and does not recover OpenVLA-OFT end-to-end. (3) Protecting one 28,672-parameter output projection restores OpenVLA-OFT near-baseline W3 success on LIBERO-Long (8.0% to 93.5%) and A8 success across all four suites. Real-kernel profiling shows all quantized paths slower than BF16 at batch size one; a 1,200-rollout Franka study reports precision comparisons with separately fine-tuned policies. Limitations: only four models, only X-VLA beyond LIBERO, no matched implementations of VLA-specific PTQ methods, ten task clusters limit bootstrap power, and the paper's own remedies are not validated on the robot.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under uncalibrated W4A4 round-to-nearest quantization, expanding π0.5's quantized action-head subset from 126 to 167 layers raises LIBERO-Spatial success from 7.0% to 70.5%, and matched-observation replay confirms the corresponding numerical interaction before environment feedback.
- Two-episode smoothing-and-clipping calibration removes severe joint failures in the tested π0.5 subsets but lowers π0 success (e.g., LLM-only 72.0% to 11.0%) and does not recover OpenVLA-OFT end-to-end.
- Protecting OpenVLA-OFT's single 28,672-parameter output projection restores near-baseline success: 93.5% W3 on LIBERO-Long (from 8.0%) and protected W4A8/W8A8 within 2.5 pp of baseline across all four suites.
- Nearly identical activation kurtosis (834.3 vs 835.3) accompanies an 81.5-pp success gap on OpenVLA-OFT, so local activation statistics alone miss task-level sensitivity.
- At batch size one on an RTX 4090, all tested quantized kernels reduce peak memory but run slower than BF16 for OpenVLA-OFT's parallel action-decoding workload.
Strengths
Controlled intervention design at unusual scale
409 runs / 94,574 episodes with nested scope interventions (Fig. 1, Tables 4, 29): isolated groups, unions, and supersets measured separately, which is the only design that can expose both the union collapse (attention+adaRMS -1.0/-5.5 pp isolated vs -97.5 pp joint) and the 126-to-167-layer recovery (7.0% to 70.5%).
Numerical evidence decoupled from the environment
Same-observation, same-noise replay over 20 held-out trajectories / 219 chunks (App. L, Table 33) shows the W4A4 interaction precedes closed-loop feedback: median error-norm ratio 2.90, residual 0.82, and chunk MAE 0.514 to 0.390 with improvement on 99.1% of chunks — foreclosing the objection that non-additivity is a mere success-thresholding artifact.
Constructive remedy with replication and a statistics counterexample
Exceptional provenance honesty
Deployment-relevant negative results
Weaknesses
Narrow architecture and benchmark coverage
RTN anchor only; no matched VLA-specific PTQ baselines
Headline effect lives in a configuration no one deploys
Limited uncertainty resolution for decisive cells
Remedies untested on hardware
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VLAQuantBench is a controlled evaluation of post-training quantization (PTQ) for vision-language-action models, motivated by the observation that the deployment-relevant outcome is closed-loop task success over an execution trajectory, not single-forward-pass error. The corpus spans 409 evaluation runs and 94,574 simulation episodes: OpenVLA-OFT, π0, π0.5, and X-VLA on all four LIBERO suites, with X-VLA extended to SIMPLER, CALVIN, and VLABench (§3). The central claims are that (i) closed-loop quantization sensitivity is non-monotone and non-additive in the quantized scope; (ii) calibration is part of the configuration and its effects do not transfer across models; (iii) a targeted remedy — protecting OpenVLA-OFT's output projection — recovers near-baseline low-bit success; and (iv) local activation statistics do not predict task sensitivity. The paper deliberately anchors on one shared RTN quantizer to isolate format, scope, and calibration effects rather than to rank PTQ methods.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VLAQuantBench?
VLAQuantBench is a controlled benchmark for measuring how post-training quantization (reducing the numerical precision of a neural network without retraining) affects the task success of vision-language-action robot models in closed-loop simulation. It comprises 409 evaluation runs and 94,574 simulation episodes across four released models (OpenVLA-OFT, π0, π0.5, X-VLA), mostly on the LIBERO manipulation benchmark, plus real-kernel and physical-robot measurements.
What is the paper's main finding?
Quantization effects on robot task success are neither monotone nor additive. Under uncalibrated W4A4 quantization, quantizing MORE action-head layers of π0.5 raised success from 7.0% to 70.5%, and isolated per-layer tests badly mispredicted joint outcomes (-97.5 pp together versus -1.0/-5.5 pp separately). Separately, keeping one tiny 28,672-parameter output projection of OpenVLA-OFT at full precision recovered near-baseline success at W3 and 8-bit-activation settings.
Does quantization make VLA robot models faster?
Not in this paper's measurements: at batch size one on an RTX 4090, all tested quantized kernels (AWQ, NF4, LLM.int8(), SmoothQuant) cut peak memory roughly in half but were slower than BF16 for OpenVLA-OFT's parallel action decoding (e.g., 139.9 ms BF16 versus 152.6-379.5 ms full inference), and the finding replicated across three sessions.
Is the code and data available?
Yes. The paper states that evaluation code, model configurations, scope definitions, episode records, and analysis scripts are publicly released at github.com/jiuyixu25/VLAQuantBench, with result-file manifests including SHA-256 hashes and recorded checkpoint revisions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as extracted text fragments; whisker and axis details were reconstructed from captions rather than inspected visually.
- This is an arXiv v1 preprint (September 2026) and has not undergone journal/conference peer review.
- The extraction includes all referenced tables read by the reviewer (1-35), but a few entries (e.g., Table 31 consolidation) could not be traced to raw result files.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.