When Faster VLA Deployment Changes Closed-Loop Behavior: Task Success-Latency Analysis of SmolVLA Across PyTorch and ONNX Variants
cs.RO · cs.LG
Quick verdict
ONNX export halves SmolVLA latency on RTX 2060 but cuts LIBERO Spatial success (70%→41%), largely from 16-token truncation; honest, careful, yet single-seed.
Summary
This letter empirically studies whether deploying the SmolVLA vision-language-action model via ONNX Runtime preserves closed-loop task behavior. On a single RTX 2060 (6 GB), the author evaluates HuggingFaceVLA/smolvla_libero in LIBERO Spatial and Object under a pinned protocol (MuJoCo 3.3.2, LeRobot 0.6.1, seed 42), comparing PyTorch+AMP against ONNX Runtime CUDA Execution Provider over 100 episodes per suite, plus a 300-episode paired rollout. PyTorch+AMP reaches 70.0% Spatial / 88.0% Object success at 1181 ms p99; requested-FP16 and requested-INT8 ONNX exports halve latency (601/532 ms p99) but Spatial falls to 41.0%/40.0% while Object stays at 89.0%. A graph audit shows both ONNX artifacts are byte-identical FP32 graphs, so the INT8 request was never applied. A 16/24/32-token language-width ablation shows width 24 recovers Spatial to 75.0% at roughly stable Object success and latency, identifying instruction truncation as a major cause. Limitations include a single seed, one GPU, two of four LIBERO suites, three non-interchangeable latency benches, and a baseline 26 points below the published 96% Spatial rate.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ONNX/CUDA-EP deployment of SmolVLA halves p99 latency versus PyTorch+AMP on an RTX 2060 (1181 → 601/532 ms) while dropping LIBERO Spatial success from 70.0% to 41.0%/40.0% and leaving Object at 89.0% (100 episodes/suite, seed 42).
- The default 16-token static language width is a major contributor to the Spatial gap: re-exporting at 24 tokens recovers Spatial success to 75.0% with Object (90.0%) and uniform-bench p99 (587.7 ms) approximately stable; width 32 yields 71.0%.
- The requested-INT8 ONNX artifact is byte-identical (same SHA-256) to the requested-FP16 FP32 graph with no quantization operators, so precision flags must be verified by graph audit rather than trusted.
- Open-loop numerical agreement does not imply closed-loop success: an early adapter reached ~0.98 per-step cosine similarity versus PyTorch yet scored 0% closed-loop success.
Strengths
Artifact-level honesty about fake quantization
Section III-C: SHA-256 audit shows requested-FP16 and requested-INT8 artifacts are byte-identical FP32 graphs (22,514 nodes, FLOAT initializers, no quantization ops), and the 601-vs-532 ms delta is reframed as CUDA-EP run variability rather than an INT8 speedup — a verification step most deployment papers skip.
Genuinely controlled width ablation with localized effect
Section III-D holds checkpoint, tether 0.12.0, opset 19, CUDA EP, hardware, seed, and protocol fixed while varying only token width; Table IV's fixed width-16 truncation grouping shows recovery concentrated where predicted (previously-truncated tasks 3-4/50 → 36/50 at width 24; untruncated stay 36-39/50).
Disciplined statistical framing
Clean separation of open-loop, latency, and closed-loop evidence
Weaknesses
Single seed for all headline comparisons
Unattributed 26-point baseline gap
Latency claims ride on three non-interchangeable benches
Structurally asymmetric suites and coarse width grid
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This v2 letter (arXiv:2609.14146) is a single-author deployment audit of HuggingFaceVLA/smolvla_libero on one RTX 2060 (6 GB), asking whether the standard "export to ONNX for speed" path preserves closed-loop task success. The pinned protocol is LIBERO Spatial/Object under MuJoCo 3.3.2, LeRobot 0.6.1, seed 42, with a 100-episode/suite main evaluation and a separate 300-episode/suite paired PyTorch-ONNX rollout. Four claims are staked: (1) the ONNX/CUDA-EP configuration roughly halves p99 latency (1181 → 601/532 ms) yet drops Spatial success from 70.0% to 41.0%/40.0% while Object stays at 89.0%; (2) the default 16-token static language width is a major contributor — re-exporting at 24 tokens recovers Spatial to 75.0% with Object (90.0%) and uniform-bench p99 (587.7 ms) approximately stable; (3) requested precision flags can be vacuous — the requested-INT8 artifact is byte-identical (same SHA-256) to the requested-FP16 FP32 graph; (4) open-loop numerical agreement does not imply closed-loop success, demonstrated by an early adapter at ~0.98 per-step cosine scoring 0%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does converting a VLA robot policy to ONNX change its task success rate?
It can. In this study, SmolVLA evaluated in LIBERO Spatial dropped from 70.0% success under PyTorch+AMP to 41.0% under a default ONNX export, despite roughly half the inference latency — mostly because instructions were statically padded/truncated to 16 tokens. Re-exporting with a 24-token width recovered Spatial success to 75.0% while keeping latency nearly unchanged. Object-suite success (short instructions) stayed at ~89% throughout.
Was the INT8 ONNX model in this paper actually quantized?
No. A graph audit (matching SHA-256 hashes, 22,514 nodes, FLOAT initializers, no quantization operators) showed the requested-INT8 artifact was byte-identical to the requested-FP16 FP32 graph. The export tool's --precision flag is consumed by an unused TensorRT path, so the 'INT8' row measures a second FP32 run, not quantization.
What latency does SmolVLA achieve on an RTX 2060?
The paper reports p99 latency of 1181 ms for PyTorch+AMP, 601 ms and 532 ms for the two ONNX CUDA-EP artifacts on the tether inspect bench, and 583.6-591.3 ms across token widths on a uniform ONNX Runtime bench. These benches are not mutually controlled, so the ~2× speedup characterizes the coupled configuration rather than a single change.
Is the width-24 ONNX model as good as the PyTorch baseline?
Its Spatial success was 75.0% versus the PyTorch+AMP baseline's 70.0%, with overlapping Wilson 95% confidence intervals and a two-proportion chi-squared p=0.53. The authors explicitly state this does not establish equivalence — only that the large width-16 deficit (41.0%) is largely removed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only as flattened text descriptions; visual details (axis ranges, point placement) could not be independently verified.
- Tables were extracted as flattened text; values appear internally consistent across abstract, tables, and conclusion, but column alignment was reconstructed from context.
- Several cited works (refs 4, 5, 6, 8, 9, dated 2025-2026) and the GitHub repository could not be verified against external sources in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.