Robot World Models Are Not Invariant to How the Actions Are Written
cs.RO · cs.LG
Quick verdict
World models collapse (2.6–13.4×) when actions switch absolute↔delta; solid diagnosis and repair, but 3 seeds and one architecture limit scope.
Summary
This paper asks whether latent world models are invariant to the action parameterization a policy happens to use — absolute joint targets versus deltas relative to the current state, two encodings of the same commanded trajectory. Training a decoder-free predictor (frozen I-JEPA ViT-H/14 latents, MLP head, InfoNCE) on one encoding and evaluating on the equivalent one collapses performance across three LeRobot datasets and two morphologies: retrieval rank worsens 2.6–13.4×, goal-conditioned action selection falls from 53.1% to 15.4%, and on PushT the predictions under the two encodings have mean cosine 0.067 (worst −0.377). A parity-plus-reconstructibility test, grounded in an MDL analysis, certifies the axis as a genuine re-parameterization (mutual reconstruction R²=0.996 on the joint input versus 0.757 on the action channel alone) and rejected three of the authors' four candidate axes. The repair averages the training objective over both encodings with a consistency penalty: held-out rank improves 61.5 to 5.19 and worst-case agreement 0.779 to 0.995, at roughly three points of canonical action-selection accuracy; Proposition 3 shows output averaging has no Jensen guarantee for direction-valued predictions. The defect is flat in data scale. Limitations: offline evaluation only, three seeds, a single frozen-encoder architecture family despite the general title, a linear admissibility probe, and no comparison against trivial inference-time input canonicalization.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 4
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A latent dynamics model trained on one action encoding (absolute joint targets) and given the identical commanded trajectory in the equivalent encoding (deltas) collapses: retrieval rank worsens 2.6–13.4× across three LeRobot datasets and two morphologies, and goal-conditioned action selection falls from 53.1% to 15.4%.
- The collapse is not information loss: the two encodings are mutually reconstructible at R²=0.996 on the joint input (versus a misleading R²=0.757 on the action channel alone), certified by a parity-plus-reconstructibility test that also rejected three of the authors' four proposed axes.
- Averaging the training objective over the two encodings restores task performance (held-out rank 61.5→5.19), and adding a consistency penalty raises worst-case cross-encoding agreement from 0.779 to 0.995 at roughly three points of canonical action-selection accuracy; output averaging has no Jensen guarantee for direction-valued predictions (Proposition 3).
- The defect is invariant to data scale (collapse ratio grows from 2.5× to 9.9× only because the canonical model sharpens while cross-encoding cosine stays flat at ~0.34–0.37), and it appears exactly when the representation does not already discharge the distinction (control axes show dispersion at 10⁻¹⁵–10⁻¹², §5.3).
Strengths
Novel and well-motivated axis of study
§1 and §5.1 identify that the absolute-vs-delta choice is made for controller-interface reasons and inherited silently by action-conditioned world models, while the visual-invariance literature audits only crops, jitter and camera pose. The axis is not contrived: both parameterizations appear in widely used codebases (Zhao et al. 2023; Chi et al. 2023 via LeRobot).
Certification methodology with a priced theory
Theorem 2 makes the parity gap δ a lower bound on the cost of enforcing orbit-constancy; the joint-input evaluation rule catches a concrete false negative (action channel alone R²=0.757 vs R²=0.996 jointly, §3/§5.1). The test rejected three of the authors' own four candidate axes (photometric jitter, camera choice, action-window summaries), which is genuine falsification discipline rather than post-hoc justification.
Honest self-accounting of what each repair component buys
Proposition 3 is a crisp, verifiable counterexample
Strong reproducibility infrastructure
Weaknesses
Statistical support is thin and below the paper's own standard
Scope narrower than the title
Missing trivial baseline: inference-time input canonicalization
Offline evaluation only
PushT headline rests on a weak, possibly broken model
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks one question: does an action-conditioned latent world model care whether the commanded trajectory is written as absolute joint targets or as deltas relative to the current state? The two encodings are exactly inter-derivable given the state (a_rel = a_abs − s_t), so any failure cannot be attributed to information loss. The title claims a general defect of robot world models; the experiments instantiate it for one family — a frozen I-JEPA ViT-H/14 encoder with a small MLP predictor trained by InfoNCE (decoder-free, joint-embedding). The authors are explicit (§1, §5.1) that reconstruction-based models such as Dreamer would partly mask action-channel defects, so the general title is broader than what is measured. Central claims: (1) cross-encoding collapse — retrieval 2.6–13.4× worse (Table 1: cabinet 9.9×, coffee 13.4×, PushT 2.6×), action selection 53.1%→15.4%, PushT cosine 0.067 with worst case −0.377; (2) a certification test (Theorem 2: parity necessary with the gap δ as an exact price; mutual reconstructibility sufficient) that rejected three of the authors' four proposed axes; (3) loss-side orbit averaging repairs accuracy (held-out rank 61.5→5.19) and a consistency penalty buys worst-case agreement 0.779→0.995 at roughly three points of canonical accuracy, while output averaging has no Jensen guarantee for direction-valued predictions (Proposition 3); (4) the defect is flat in data scale (§5.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Robot World Models Are Not Invariant to How the Actions Are Written' show?
It shows that a latent world model trained to predict future states from actions written one way (absolute joint targets) fails badly when the identical commands are written the other way (deltas relative to the current state): retrieval quality drops 2.6–13.4× across three robot datasets, goal-based action choice falls from 53.1% to 15.4%, and on PushT the model's two predictions for the same future are nearly orthogonal (cosine 0.067).
Isn't this just ordinary distribution shift?
The authors argue no: the two action encodings carry exactly the same information — they reconstruct each other at R²=0.996 when the current state is included (only R²=0.757 from the action channel alone) — so the failure is brittleness to a notation change rather than a justified response to missing information. They provide a test, predictive parity plus mutual reconstructibility on the joint input, to certify when a transformation is a mere re-labeling.
How is the problem fixed?
By averaging the training loss over both encodings, optionally with a consistency penalty. This restores accuracy (held-out retrieval rank 61.5 to 5.19 on ALOHA cabinet) and raises worst-case cross-encoding agreement from 0.779 to 0.995, at roughly three points of canonical action-selection accuracy. Averaging model outputs instead does not work for direction-valued predictions — Proposition 3 gives a counterexample where the normalized mean scores below every member of the orbit.
What data and code does the paper use?
Three public LeRobot datasets (ALOHA mobile cabinet, ALOHA static coffee, PushT), a frozen I-JEPA ViT-H/14 encoder, and a small MLP predictor trained with InfoNCE. The authors release their codebase (Mezzanine) with cached latents, listed seeds and commands, and stored result files; a full three-seed sweep reportedly takes minutes on a single RTX 5090 once latents are cached.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 could not be visually inspected (text extraction only); the qualitative frame evidence of Figure 1 is taken on trust from its caption.
- The Mezzanine repository release is claimed in the reproducibility statement but could not be verified from the text.
- Some appendix proofs (e.g., Theorem 5, Appendix E) are asserted as routine rather than shown; assessment of the MDL framework relies on the statements as given.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.