Reinforcement Learning for Real-Time Vision-Language-Action Policies
Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn
cs.RO · cs.LG
Quick verdict
Latency-aware RL fine-tuning of VLAs shows impressive sim and real-world gains, though headline claims somewhat outrun its statistical reporting.
Summary
The paper proposes Real-Time EXPO-FT, a reinforcement-learning fine-tuning framework for vision-language-action (VLA) policies that must act under nontrivial inference latency. Building on EXPO-FT (off-policy RL with a learned edit policy atop a VLA) and training-time real-time chunking, it decouples slow asynchronous generation of candidate action chunks by the large VLA from a fast, synchronous step that edits candidates conditioned on the latest observation and selects the highest-Q chunk; a noise-space filter critic keeps Bellman backups cheap with many candidates. On 10 Kinetix tasks under a 4-step delay it reaches 96.2% average success versus 61.7-81.7% for delayed baselines, and exceeds zero-delay RLPD (81.4%). On four real-world dynamic tasks (object passing, ball balancing, soccer kicking, dynamic picking) at 30 Hz with ~100-167 ms latency and at most 10 minutes of online data, it attains 29/30 average success versus 25/30 for the strongest baseline. Limitations: human resets, per-task hand-crafted success detectors, privileged critic inputs on two tasks, simulation uses state-based policies without an actual VLA, no confidence intervals or significance tests, and the 'best in 10/10 environments' claim is not consistent with the extracted Table II, where some baselines are nominally higher on individual tasks.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under inference latency, standard RL fine-tuning of VLAs suffers a non-Markovian distribution shift, and explicitly decoupling slow asynchronous chunk generation from fast, latest-observation-conditioned action edits plus Q-based selection fixes this.
- On 10 Kinetix tasks under a 4-step inference delay, Real-Time EXPO-FT achieves 96.2% average success, outperforming all delayed RL baselines and even the zero-delay RLPD baseline (81.4%).
- On four real-world dynamic manipulation tasks at 30 Hz, capping online data at 10 minutes, Real-Time EXPO-FT improves average success from 42% (SFT) to 97% (29/30) without human intervention during training.
- Noise-space candidate filtering during Bellman backups enables large candidate sets (N=32) at low compute, accelerating training relative to denoise-then-filter alternatives.
Strengths
Clear formulation of a genuinely underexplored problem
Section IV-A explicitly identifies that under inference latency the executed action is a function of s_{t-d}, making the delayed process non-Markovian and biasing standard RL credit assignment; prior real-time execution work (RTC, Training-Time RTC, VLASH, pi-R2) is imitation-based and offers no RL improvement mechanism.
Strong simulation results under delay
Table II: 96.2% average success across 10 Kinetix tasks with a 4-step delay, improving 34.5/20.1/21.3/14.5 points over DSRL, DSRL w/ RTC, EXPO-FT, EXPO-FT w/ RTC, and surpassing even the zero-delay RLPD baseline (81.4%).
Sample-efficient real-world demonstration on genuinely dynamic tasks
Compute-aware training design with ablation
Unusually detailed implementation appendix
Weaknesses
Headline 'best in 10 out of 10 environments' claim conflicts with Table II
No statistical inference anywhere
Simulation does not actually involve a VLA
Methodology inconsistency between sim and real
Reward-engineering creep undercuts the 'sparse reward' framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does Real-Time EXPO-FT solve?
Large vision-language-action (VLA) robot policies take around 100+ milliseconds to infer an action chunk, so by the time actions execute the observation they were computed from is stale, which degrades performance on fast, dynamic tasks. The paper combines reinforcement-learning fine-tuning with asynchronous execution: the slow VLA generates candidate action chunks in the background while a small edit policy reacts to the newest observation and a learned Q-function picks the best chunk just before execution.
What results does the paper report?
On 10 Kinetix simulation tasks with a 4-step inference delay it averages 96.2% success, above all delayed baselines and even a zero-delay RLPD baseline (81.4%). On four real-world dynamic manipulation tasks at 30 Hz (object passing, ball balancing, soccer kicking, dynamic picking) it reaches an average of 29/30 successes using at most 10 minutes of online robot data, versus 25/30 for the strongest prior method.
What robot and base model are used in the real-world experiments?
A single-arm DROID setup controlled in end-effector space at 30 Hz with one side camera and one wrist camera. The base policy is pi-0.5 fine-tuned with LoRA, with total latency of roughly 100-167 ms simulated per task. Rewards are sparse binary signals from rule-based success detectors, and training runs without human intervention aside from environment resets.
What are the main limitations of the approach?
A human performs environment resets, each task needs a hand-crafted success detector, two tasks give the critic privileged state such as ball or defender position, the simulation benchmark uses state-based policies rather than actual VLAs, and the paper reports no confidence intervals or significance tests, so some per-task margins are hard to statistically support.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table extraction is column-misaligned (the Table II average row has nine values under eight visible column headers, and the text-implied DSRL delay-0 column is missing), so per-task numbers cited here may be shifted by one column; cross-checks used text-reported averages.
- Figures 1-10 are not analyzable from the extracted text; judgments rest on captions and in-text numbers only.
- The metadata abstract is truncated mid-sentence; the embedded full text, which appears complete through the appendices, was used instead.
- This is a September 2026 arXiv preprint (2609.18207); peer-review status is unknown.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.