One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling
Geigh Zollicoffer, Minh Vu, Rajiv Ranasinghe, Manish Bhattarai
cs.AI
Quick verdict
Audits a multimodal world model: text answers all 22 physics probes correctly but video ignores parameters; rigorous, honest, though only one model family.
Summary
The paper audits whether a unified multimodal world model (NVIDIA Cosmos3-Super) tells one coherent physical story across modalities. It formalizes internal misalignment (generated video vs. the same checkpoint's text prediction) and external misalignment (video vs. an analytic/Isaac Sim physics reference), scoring both with shared 'predictive contracts' — event class, magnitude in object-relative units, keyframes — extracted from pixels by a validated RGB tracker. Across four mechanisms (bounce, incline, push, pendulum), 20 parameter settings and eight seeds, the text pathway answers all 22 physics probes correctly, yet neutrally prompted videos rarely realize the stated dynamics: incline stays frozen in all 24 slide-condition clips, push renders a parameter-blind ~10-cube-width slide, and pendulum over-amplifies 1.2–2×. Tiered interventions (A ladder: the model's own contract; B ladder: corrected physics) and wrong directives (W1/W2) show language controls whether the binary event occurs — in both directions — but not its magnitude, and richer specifications sometimes increase degeneration. A second checkpoint (Nano) shows a shared failure family. Strengths include preregistered tolerance rules, sensor-validation checks (R²=0.989/0.997) and candid reporting of a superseded pilot. Key limitations: a single model family, eight seeds with wide binomial intervals, an evaluator whose incline camera exposes only 4.5 of 15 cube-widths, and admitted under-powered A/B ladder comparisons.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cosmos3-Super's text pathway answers all 22 physics probes correctly, while its neutrally prompted video generation frequently fails to realize the same dynamics, making internal and external misalignment coincide.
- Language directives control whether the binary physical event occurs in generated video — in both correct and deliberately wrong directions — but do not control event magnitude.
- The A ladder (model's own contract) and B ladder (corrected physics) do not consistently close the internal or external gaps; only incline shows dose-response, saturating at the evaluator's visible ceiling.
- Generated video is largely insensitive to the physical parameter value itself (class-constant seeds: incline 8/8, bounce 6/8, push 7/8) while remaining sensitive to event-naming words.
- The failure pattern is shared across checkpoints, with Cosmos3-Nano reproducing the parameter-blind push and over-amplified pendulum.
Strengths
Conceptually novel two-reference design
§3/Fig. 3 compares the same measured video contract C_video against two references — the checkpoint's own text contract (D_int) and analytic/Isaac physics (D_ext) — separating consistency from correctness, which §2 correctly argues external-only benchmarks (VideoPhy, WorldModelBench) cannot express.
Crossed wrong-directive arms establish causality of language control
W1 (qualitative-wrong) and W2 (magnitude-wrong) hold parameter, conditioning frame, and duration fixed (§4): pendulum W1 drops dynamics to 3/8 vs. 6–7/8 under correct tiers; incline W2 collapses displacement to 1/8 (Table 2). This cleanly rules out parameter recall as the tier effect's cause.
Above-average scoring discipline
Transparent superseded-pilot reporting
Second-checkpoint replication and oracle analysis
Weaknesses
Single model family undercuts the general conclusion
Eight-seed cells with no formal tests on the central contrasts
Evaluator constraints confound three of four axes
Ladder questions (Q3/Q4) are framed but unpowered
Loose probe accounting behind the 22/22 headline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether a unified multimodal world model — NVIDIA Cosmos3-Super, a Mixture-of-Transformers checkpoint with separate 31.2B understanding and generation streams plus ~1.6B shared parameters (§4, A.1) — tells one coherent physical story across modalities. It formalizes two failure modes through shared predictive contracts C = (event, magnitude, units, keyframes, invariants) (§3): internal misalignment D_int between generated video and the same checkpoint's text prediction T_text, and external misalignment D_ext between video and an analytic/Isaac physics reference T_phys. Four mechanisms (bounce/restitution, incline/Coulomb friction, push/kinetic friction, pendulum/timing), 20 parameter settings, eight seeds; 160 neutral clips, 192 tiered clips, a 32-clip paired A/B arm — 448 scored clips in total. Central claims: (1) text is exactly correct on all 22 physics probes while neutral video rarely realizes the stated dynamics; (2) language controls binary event occurrence bidirectionally but not magnitude; (3) the A/B intervention ladders do not consistently close the gaps.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between internal and external misalignment in world models?
In this paper, internal misalignment is when a model's generated video disagrees with the same model's own text prediction for the same scenario; external misalignment is when the video disagrees with the true physics, given by an analytic solution or an Isaac Sim simulation. A video can match the text yet be physically wrong, or match physics while contradicting the model's own stated answer.
Which world model was audited in this study?
NVIDIA Cosmos3-Super, a Mixture-of-Transformers checkpoint with separate 31.2B-parameter understanding and generation streams plus roughly 1.6B shared parameters. A smaller Cosmos3-Nano checkpoint served as a robustness check. Scenes ran in Isaac Sim 6.0.1 (PhysX) covering bounce, incline friction, push, and pendulum mechanics.
What did the study find about language control over generated physics?
Text prompts could switch binary outcomes in both directions — keyframed prompts unlocked incline sliding in 6 of 8 seeds, while a false 'it does not swing' directive suppressed pendulum swings in 5 of 8 seeds — but the magnitude of the event never followed commanded numbers: push travel stayed near a fixed ~10 cube-widths regardless of the stated speed, and pendulum amplitude stayed 1.2–2× too large.
Does the paper release code or data?
The extracted text states no code or data release. It does specify scenes, seeds, tolerance rules, and inference parameters in unusual detail, so re-running is possible in principle, but it requires the Cosmos3 checkpoints, Isaac Sim, and substantial GPU capacity.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract and a few sentences are truncated in extraction ('often in disagreeme...'); conclusions were cross-checked against §7.
- Figures 1–5 are schematics/parameter-sweep plots not analyzable from text; sweep-shape claims rest on prose and Table 1 counts.
- Tolerance is stated as max(0.5 units, 25%) in §3 but ±30% in Table 2's caption — a minor protocol inconsistency I could not resolve from the text.
- Appendix-referenced materials (tracker implementation, per-clip scores, A.1 full audit) are only summarized; claims about them are taken on trust.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.