Sampling headroom is not selection gain: a compute-value audit of test-time scaling for video world models
Yuhua Jiang, Junjie Lu, Feifei Gao
cs.CV · cs.AI
Quick verdict
Four-gate compute-value audit of video test-time scaling; preregistered, honest negative results, but its headline deployable selector is post hoc.
Summary
This paper introduces the Compute-Value Audit (CVA), a sequential framework that decomposes test-time scaling (TTS) value into four ordered gates — opportunity, state confidence, action value, and entry fee — and applies it to video world models. On 192 Physics-IQ scenes, widening the candidate pool from 4 to 16 raises oracle quality by +9.23 IQ (95% CI [+7.44, +11.14]), yet flow-residual, cycle-consistency, and VideoReward verifiers fail to recover this headroom reliably. Across three generators (Wan2.2-5B, ABot-PhysWorld-14B, Cosmos3-Nano), none of twelve adaptive-depth policies beats uniform compute, recovering only 42-69% of the measured entry fee, and a matched-60-NFE Predict-and-Perturb intervention is negative on VideoPhy2 across three fresh-seed replicas. The audit is not vacuously negative: anchor-explorer clears all four gates on sparse PRM800K, MMLU-Pro isolates a state-versus-action failure, and a privileged paired future yields positive video gains as an upper bound. A Gaussian Verifier-Quality Law ties recovered headroom to verifier-target correlation (held-out R2=0.629). Key limitations: the best deployable selector (CVA-Select) was chosen post hoc from frozen component rows without confidence intervals, PAI gains are small, the OpenS2V transfer is negative, RoboTwin gains do not reproduce, and the PRM800K pass rests on 7/128 problems. Code is released; full replication requires substantial GPU compute.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On 192 Physics-IQ scenes, expanding the pool from 4 to 16 candidates raises oracle IQ by +9.23 (95% CI [+7.44, +11.14]), but flow-residual, cycle-consistency, and VideoReward verifiers fail to convert this headroom into reliable selection gain.
- Across three generators, none of twelve adaptive-depth policies outperforms uniform compute at matched mean depth; the best recover only 42-69% of the measured entry fee (Appendix B, Table 13).
- A matched-60-NFE Predict-and-Perturb intervention on VideoPhy2 is negative across three fresh-seed replicas (primary -1.68 pp Joint; 0/3 nonnegative).
- The audit is not vacuously negative: anchor-explorer clears all four CVA gates on sparse PRM800K, and a privileged paired future establishes a positive video upper bound (+4.80 IQ from B=4 to 16, Table 5).
- Recovered headroom is priced by verifier-target correlation, consistent with a Gaussian Verifier-Quality Law (held-out R2 = 0.629, cross-fitted).
Strengths
Compute-complete accounting contract
The one-candidate-budget rule (Sec. 3.1) forces every selector to score exactly B generated candidates, and Appendix M restates the invariant; Table 23 gives a per-scene ledger separating generation, same-model verification, and external RM calls, so no route hides generation cost.
Preregistration and provenance discipline
Sealed preregistration IDs (e.g., 76a721b3, 5b4d5d75), SHA-256 result hashes for the P&P runs (Appendix C) and analyses (Appendix H), frozen pick files opened only after outcomes — rare and verifiable practice for an empirical paper.
Constructive controls prevent blanket negativity
Candid status labeling of its own best method
Theory tied to measurement
Weaknesses
Headline deployable selector is post hoc and uncertainty-free
Positive control is fragile and sparse
PAI transfer claim is thinner than framed
Minimal Gaussian bridge from theory to empirics
Benchmark concentration limits generality of the negative headline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks when extra inference compute for frozen video world models produces deployable gain rather than merely better candidates in aggregate. It contributes the Compute-Value Audit (CVA) — four ordered gates (opportunity, state confidence, action value, entry fee) — plus two compact theories: a Verifier-Quality Law (Prop. 1: under a Gaussian score-quality model, the selected candidate's expected quality is rho times the oracle maximum) and a Memory Value Law (Prop. 2: memory value = addressable heterogeneity - retrieval regret). Headline empirical claims: (1) on 192 Physics-IQ scenes, oracle IQ rises +9.23 (95% CI [+7.44, +11.14]) when pools grow 4 to 16; (2) flow/cycle/VideoReward verifiers fail to recover that headroom; (3) twelve adaptive-depth policies on three generators recover only 42-69% of the entry fee and never beat uniform; (4) a matched-60-NFE Predict-and-Perturb intervention is negative across three fresh-seed replicas on VideoPhy2 (primary -1.68 pp Joint); (5) positive controls (PRM800K anchor-explorer; privileged paired future) show the audit certifies value when the information exists.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Compute-Value Audit (CVA) in this paper?
CVA is a four-gate sequential check for test-time scaling: (1) opportunity — does extra sampling create oracle headroom over the incumbent; (2) state — do observable signals carry reliable information; (3) action — does acting on that signal improve quality; (4) entry fee — does the gain exceed the full cost of generation plus verification. It diagnoses where a scaling intervention stops producing deployable value.
Does test-time scaling help video generation, according to this paper?
In the tested settings, no deployable method reliably converted wider sampling into better selected video. Oracle headroom existed (+9.23 IQ when going from 4 to 16 candidates on 192 Physics-IQ scenes), but flow, cycle-consistency, and VideoReward verifiers failed to recover it, twelve adaptive-depth rules never beat uniform compute, and a compute-matched refinement loop (Predict-and-Perturb) was negative across three fresh-seed replicas.
Is the paper entirely negative about test-time scaling?
No. It includes positive controls: an anchor-explorer policy clears all four audit gates on a sparse 128-problem PRM800K math set, MMLU-Pro shows the audit can pinpoint state-versus-action failures, and a privileged paired-future signal achieves positive selection gains in video, proving the missing ingredient is information, not possibility.
Is code or data available for this paper?
The abstract states code is available at github.com/YuhuaJiang2002/sampling-headroom-is-not-selection-gain, and the appendix binds reported numbers to frozen pick files, sealed preregistrations, and SHA-256 result hashes. Full replication requires generating candidate pools with three 5B-14B video models on public benchmarks, which needs substantial GPU compute.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images are absent from the extraction (only Fig. 1 and Fig. 2 captions are present); all reviewed numbers come from tables and body text, which appear complete through Appendix O and Tables 7-28 with no end-of-text truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.