Outcome-Conditioned End-Effector Geometry Across Vision-Language-Action Policies
Xingyu Lin, Zhuang Li, Zhongrun Wu, Shouquan Zhou, Dehui Du
cs.RO · cs.AI
Quick verdict
The paper shows both-succeeding VLA policies trace more similar end-effector paths; a rigorously controlled, honest, but simulation-only descriptive study.
Summary
Does a vision-language-action (VLA) policy that succeeds trace the same physical path as another policy that succeeds on the same task? Using 15,000 closed-loop LIBERO rollouts from four policies (OpenVLA, OpenVLA-OFT, UniVLA, pi0.5), the authors build 3,600 configuration-matched policy pairs at the clean visual level and measure normalized DTW distance between end-effector paths, stratified by joint outcome. Both-success pairs have a median distance of 0.0120 m versus 0.0380 m when exactly one succeeds; the ordering holds in every task, policy pair, and across nine sampling and band-limited representations, though the ratio varies from 2.53 to 6.13, so only the direction is stable. Both-failure pairs are more separated but rest on thin support (89 pairs, 64 from one policy combination) and are reported as exploratory. Matched controls show that replacing the initial state within a task barely increases distance (0.0157 m) while changing task does (0.0817 m), and a within-policy baseline still reveals small residual policy differences. A fixed 72-action window and endpoint/duration regression adjustments shrink but do not eliminate the gap. Limitations: simulation-only evidence, a position-only metric, and code/data not yet released.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
4 strengths · 4 weaknesses · 0 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Both-success policy pairs have a median normalized DTW distance of 0.0120 m versus 0.0380 m for mixed-outcome pairs, with the SS<SF ordering robust across every task, policy pair, template, and nine path representations.
- The direction of the outcome-associated gap survives band-limited alignment, a fixed 72-action window, and endpoint/duration regression adjustment, but its magnitude is representation- and specification-dependent (ratio 2.53-6.13).
- Within successful executions, agreement is largely task-associated (across-task median 0.0817 m vs 0.0157 m for a within-task state change), and a within-policy baseline still reveals small, heterogeneous residual policy differences (median paired increment 0.0021 m; repeat experiment 0.0082 m).
- Both-failure pairs are more separated (0.0439 m) but rest on thin, uneven support (89 pairs, 64 from one policy combination) and support no stable three-stratum ordering.
- Under composite visual stress, policy rankings and pair composition change together (e.g., at L4 no SS pairs exist and 3,564 of 3,600 comparisons are FF), so distances and sample composition must be read jointly.
Strengths
Outcome-conditioned design with matched controls
Configuration-matched pairs stratified by joint outcome, successful partner replacements separating state from task identity (Fig. 2b), the Eq. 3 within-policy baseline, and a demonstration reference at matched task identity together localize the effect far better than typical benchmark papers.
Alignment and sampling sensitivity systematically quantified
Nine representations including synchronous comparison and Sakoe-Chiba bands at realized fractions (18.37%/8.16%/4.08%); the 2.53-6.13 ratio range is honestly reported, with claims restricted to direction, plus a pre-specified 72-record window cohort analysis (Table V).
Statistical reporting discipline
Repeat experiment separating decoder determinism from policy difference
Weaknesses
Headline effect is partly outcome-induced mechanics
Narrow external validity and position-only metric
Artifacts not yet released; pipeline deviations from official recipes
Unexplained deviation from published numbers and OFT-heavy FF stratum
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks what success rates cannot show: when two different vision-language-action (VLA) policies both solve the same LIBERO-Spatial task from the same initial state, instruction template, and visual condition, do their end-effector paths agree? From 15,000 closed-loop rollouts of four pipelines (OpenVLA, OpenVLA-OFT, UniVLA, pi0.5), the authors form 3,600 configuration-matched unordered policy pairs at the clean level L0 and compare end-effector positions via length-sum-normalized DTW (Eq. 1-2), stratified by joint outcome (SS/SF/FF). The central claim is directional: both-success pairs have median distance 0.0120 m (n = 2,588) versus 0.0380 m for mixed pairs (n = 923), an ordering holding in all ten tasks, six policy pairs, three templates, and nine sampling/band-limited representations — while the SF/SS ratio varies 2.53-6.13, so no fixed multiple is claimed. Secondary claims: within-success agreement is largely task-associated (across-task replacement 0.0817 m vs 0.0157 m for a within-task state change); a within-policy baseline leaves a small residual cross-policy increment (median paired difference 0.0021 m; repeat experiment 0.0082 m); FF (0.0439 m, n = 89) is exploratory; and a fixed 72-action window plus endpoint/duration adjustment shrink but do not eliminate the gap.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Across 3,600 matched policy pairs on LIBERO-Spatial, pairs where both VLA policies succeed have much smaller end-effector path distances (median 0.0120 m) than pairs where exactly one succeeds (0.0380 m). The ordering is robust across tasks, policy pairs, and nine path representations, though the gap's size varies several-fold with representation.
Do the results mean VLA policies are interchangeable?
No. A within-policy baseline reveals small but measurable residual policy differences (median paired increment 0.0021 m; repeat experiment 0.0082 m), and the authors explicitly state that low cross-policy distance does not imply interchangeability.
Does this study involve real robots?
No. All 15,000 rollouts come from the LIBERO simulator (Spatial suite for four policies; Object suite for pi0.5 only). The position-only end-effector metric also excludes orientation, contact, and force, so 'similar geometry' is not 'similar execution quality'.
Is the code and data available?
The paper states code and data will be made publicly available upon publication. At review time they had not been released, and the authors note that realized corruption seeds and queried images were not saved during collection.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted captions; distribution shapes in Fig. 2-3 could not be inspected
- The abstract is truncated mid-sentence in the provided text; its final claims were cross-checked against Sections IV-E/IV-F
- Table II mask notation lost superscripts in extraction (e.g., '402' = 40-pixel side length); values cited accordingly
- References could not be verified against external sources per task constraints
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.