CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations
cs.RO · cs.AI · cs.CV
Quick verdict
Argues clean fixed-camera evaluation cannot rank visual physics inference; careful, candid MuJoCo study with controls, but single-scene, no code release.
Summary
The paper argues that the standard evaluations of pretrained visual encoders for physical reasoning — perturbation benchmarks and linear probes, almost always run in clean, fixed-camera scenes — cannot distinguish an encoder that infers physics from one that does not. The authors propose CALIPER, a calibration-conditioned prediction task in MuJoCo: an object of unknown mass and friction is struck three times; the two calibration clips are shown in full, the query only up to the moment of contact, and a linear readout on frozen features must predict the slide distance; swapping in another object's calibration verifies the evidence is used. Across 2,000 episodes and eight representations, calibration adds about +0.50 R² and the swap removes it. In the clean scene every representation lands within 0.020 R² of the simulator-state ceiling (0.992), because a fixed camera exposes displacement directly in pixel coordinates. Resampling camera, lighting, and clutter per clip spreads the same encoders across 0.498 R²; in goal-conditioned speed selection V-JEPA 2 misses by 4.0 mm while a random ViT misses by 19.6 mm, and linear probes track none of this. An appendix audits the authors' own earlier perturbation benchmark, finding 13 of 20 cells inert. Limitations: a single small simulator, a confounded nuisance scene, linear readouts only, and no stated code release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In clean fixed-camera scenes, perturbation benchmarks and linear probes cannot distinguish physics-inferring encoders from a random one: all eight representations land within 0.020 R² of the 0.992 simulator-state ceiling.
- The failure is mechanical: a fixed camera exposes displacement as a fixed pixel-coordinate pattern, so a linear readout recovers the slide from any injective feature map, raw pixels (0.996) included.
- Resampling camera, lighting, floor, and clutter per clip spreads the same eight representations across 0.498 R², and the swap control confirms the resulting ordering reflects physical inference rather than a new shortcut.
- Linear probe accuracy is uninformative about use: frame aggregation moves the random ViT's friction probe from 0.361 to 0.672 with +0.000 prediction change, and erasing the probed mass direction costs raw pixels −0.001 R² in the clean scene but −0.350 under nuisance.
- A benchmark's ranking ability must be demonstrated via three checks: a model-free effect-size audit, the spread between trivial baselines and the best model, and a swap or erasure test behind any use claim.
Strengths
Nested design with simulator-state anchors
Section 3 nests four conditions (action-only 0.524/0.514 → query → query+calibration → query+swapped calibration) against true-state ceilings (0.992 from calibration displacements, 0.988 from true (m, μ)), so every headline number has a known reference point rather than an implicit one.
The swap control turns 'evidence is used' into a measurement
A derangement assigns another episode's calibration clips; the ~+0.50 R² gain disappears in both scenes (Table 1), cleanly separating physical inference from shortcuts.
Unusual candor in Appendix A
Environment design is argued, not defaulted
Task-unit translation of abstract scores
Weaknesses
Single small simulated world carries the whole argument
The headline nuisance-scene number is confounded by design
Linear readout only, and a limited model pool
No uncertainty quantification and no code release
Critique of published benchmarks is asserted, not demonstrated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CALIPER (arXiv:2609.08250, cs.RO) is an evaluation-methodology paper with a benchmark-construction core, authored by two independent researchers. It argues that the two dominant ways of scoring pretrained visual encoders for physical competence — single-parameter perturbation sensitivity and linear probing — are run almost exclusively in clean, fixed-camera scenes, and that in that scene they cannot rank encoders at all. Five verifiable claims emerge (Sections 1 and 4): (1) in the clean scene all eight tested representations — V-JEPA 2, VideoMAE, VC-1, DINOv2, R3M, a randomly initialised ViT, frame differences, raw pixels — land within 0.020 R² of the simulator-state ceiling of 0.992; (2) the failure is mechanical, since a fixed camera turns displacement into a fixed pixel-coordinate pattern recoverable by a linear readout from any injective map; (3) resampling camera, lighting, floor, and distractors per clip spreads the same encoders across 0.498 R² while the swap control still removes every gain; (4) probe R² is uninformative about use — aggregation choice moves probes without moving prediction, and erasure cost flips from −0.001 to −0.350 R² for raw pixels across scenes; (5) benchmark ranking ability must be demonstrated via three proposed checks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CALIPER benchmark?
A MuJoCo evaluation task for pretrained visual encoders: an object of unknown mass and friction is struck twice at known speeds (calibration clips shown in full), a third strike is shown only up to the moment of contact, and a linear readout on frozen features must predict the slide distance. Swapping in another object's calibration clips tests whether the model actually used the interaction history.
Why can't clean fixed-camera scenes measure physical reasoning?
With a fixed camera, an object's displacement appears as a fixed pattern of pixel coordinates, so a linear readout can recover slide distance from almost any feature map. In the paper's clean scene, even raw pixels (0.996 R²) and a randomly initialised ViT (0.977) reach the simulator-state ceiling (0.992) without any physical inference.
Which encoder performed best on CALIPER?
Under per-clip camera, lighting, and clutter variation, V-JEPA 2 retained 0.974 R² on query-outcome prediction and missed goal distances by 4.0 mm, ahead of frame differences (0.944), VideoMAE (0.912), and raw pixels (0.861). Per-frame encoders VC-1, R3M, and DINOv2 fell to 0.67–0.71, and a random ViT dropped to 0.476, below the speed-only floor of 0.514.
What three checks does the paper propose for benchmarks?
(1) An effect-size audit: simulate the perturbation with no model and report the outcome change per cell. (2) Report the spread between random features, raw pixels, and the best model — a few hundredths means the scene, not the encoder, sets the score. (3) A swap or erasure test behind any claim that a representation uses a property, since probe decodability is neither necessary nor sufficient for use.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 1's raw-pixels clean-scene value extracts as '.0.996' (stray punctuation artifact); it is read as 0.996, consistent with the prose and Figure 2. Otherwise the extraction is complete through Appendix B and the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.