Planned Test-Time Scaling with Coordinated Reasoning Paths
Xueqing Wu, Langxing Bai, Hritik Bansal, Po-Nien Kung, Shuo Li, Hao Liu, Nanyun Peng, Kai-Wei Chang
cs.CL · cs.AI
Quick verdict
PTTS coordinates branches via a planner instead of repeated sampling; math pass@k gains up to +13.4 are promising but lack error bars and generality.
Summary
The paper proposes Planned Test-Time Scaling (PTTS), replacing independent repeated sampling with a coordinated joint policy: a planner generates k solution outlines in a single autoregressive pass, and a frozen reasoning-model executor solves the problem once per outline. The authors prove PTTS strictly generalizes repeated sampling (RS) and show, in a stylized mode-selection model, that pass@1-optimal policies collapse onto a dominant mode while a pass@k-optimal planner hedges across complementary modes, decaying residual error exponentially faster. On five math benchmarks (MATH-500, AIME 2024–2026, HMMT-Feb26) with Qwen3-1.7B/4B, PTTS-ZS (zero-shot planner) improves pass@64 over RS by up to 6.7 points, and PTTS-RL (GRPO-trained planner with 4k-token truncated execution rollouts) by up to 13.4 points (6.7 for 4B); PTTS-RL's pass@32 already exceeds RS's pass@64 at both scales (58.3 vs 55.8; 65.1 vs 63.8). Ablations support branching factor k=4, the 4k training budget, and transfer of small planners to larger executors (+4.4 pass@64). Limitations: evaluation is confined to mathematics with a single model family; no error bars, seeds, or significance tests accompany the pass@k differences; outline adherence judged by GPT-5-mini is only ~70%, leaving open whether coordination or mere prompt perturbation drives gains; the introduction misattributes a +4.4 transfer gain that §4.3 reports as +3.3.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Replacing independent repeated sampling with a planner-executor joint policy (PTTS) strictly generalizes repeated sampling, so achievable pass@k cannot be worse.
- In a stylized mode-selection model, pass@1-optimal policies collapse onto a dominant mode while pass@k-optimal planning hedges across complementary modes, yielding exponentially faster error decay (Propositions 1–3).
- Across five math benchmarks with Qwen3-1.7B/4B executors, PTTS-ZS improves pass@64 over repeated sampling by up to 6.7 points and PTTS-RL by up to 13.4 points.
- PTTS-RL's pass@32 exceeds repeated sampling's pass@64 at both model scales (58.3 vs 55.8; 65.1 vs 63.8), implying more than 2× compute efficiency.
- PTTS-RL planners transfer to unseen executors, adding +4.4 pass@64 with a larger 4B executor and remaining effective (+3.3 per §4.3) on top of a diversity-aware RL executor.
Strengths
Clean formalization with genuine, fully proved theory
§3.1–3.2 frame PTTS as a planner-executor factorization of the joint k-branch policy and prove strict generalization of repeated sampling; Appendix A gives complete proofs (Propositions 1–3) that pass@1 optimization collapses onto a dominant mode while pass@k optimization hedges, with a finite hedging threshold k*ptts and an exponential error-rate separation, instantiated in closed form for Figure 2.
Consistent gains with a practically important efficiency result
Table 1: PTTS-RL (1.7B planner) reaches 62.2 vs RS 53.3 pass@64 on the AIME average with Qwen3-1.7B executor, and 70.0 vs 65.6 with the 4B executor; per-benchmark maxima are +13.4 (1.7B) and +6.7 (4B); PTTS-RL pass@32 exceeds RS pass@64 at both scales (58.3 vs 55.8; 65.1 vs 63.8).
Informative ablations beyond headline numbers
Complementarity demonstrated, not merely claimed
High setup transparency
Weaknesses
No uncertainty quantification anywhere
~30% outline non-adherence undermines the mechanism story
Narrow evaluation scope for a 'general framework' claim
Internal inconsistency in transfer numbers
Judge-dependent metrics and unclear compute accounting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PTTS targets redundancy in test-time scaling: k independent samples from one policy concentrate on high-probability reasoning modes, so extra compute buys diminishing returns (§1). The proposed fix decomposes the joint k-branch policy into a planner (k outlines in one autoregressive pass) and a frozen executor (one solution per outline) (§3.1). The paper claims: (i) strict generalization of repeated sampling, so achievable pass@k cannot be worse (§3.2); (ii) a stylized-mode analysis in which pass@1 optimization collapses onto a dominant mode while pass@k optimization hedges, with exponentially faster error decay (§3.2, Appendix A, Propositions 1–3); (iii) empirical gains of up to +6.7 pass@64 (PTTS-ZS) and +13.4 (PTTS-RL) over repeated sampling across five math benchmarks with Qwen3-1.7B/4B (§4.2, Figures 4–5). Most claims are scoped to math and the Qwen3 family; the abstract's general-framework framing outruns the evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Planned Test-Time Scaling (PTTS)?
PTTS is an inference method for LLM reasoning that replaces drawing k independent solutions from one model with a two-stage coordinated process: a planner model first writes k different solution outlines in a single pass, then a frozen executor reasoning model solves the problem once per outline. The goal is to make the k attempts explore distinct reasoning strategies instead of repeating similar attempts.
How much better is PTTS than repeated sampling?
On five math benchmarks (MATH-500, AIME 2024–2026, HMMT-Feb26) with Qwen3-1.7B/4B executors, the zero-shot variant (PTTS-ZS) improves pass@64 by up to 6.7 points and the RL-trained planner (PTTS-RL) by up to 13.4 points over repeated sampling; PTTS-RL's pass@32 already beats repeated sampling's pass@64 at both scales (58.3 vs 55.8; 65.1 vs 63.8). Note the paper reports no error bars or significance tests for these differences.
Is code available for PTTS?
The paper links a public GitHub repository (github.com/shirley-wu/planned-test-time-scaling) and discloses prompts, training hyperparameters (GRPO via verl on DAPO-Math-17K, 8×A100, 240 steps), and analysis protocols. Zero-shot PTTS-ZS is easy to rerun with open Qwen3 weights; PTTS-RL requires planner RL training on 8×A100.
Why use a base model as the planner instead of the reasoning model?
The authors found reasoning models make poor planners: they commit early to one strategy and start solving rather than producing several diverse high-level outlines. PTTS therefore uses the pre-RL base version of the same model family as planner and the reasoning model as executor. A practical caveat: base checkpoints are not always released for proprietary models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 4–9 are available only as captions and in-text numbers in the extraction; per-benchmark curves and the Guided Sampling baseline results could not be independently verified.
- Some formulas (the pass@k estimator, the GRPO objective, Proposition 2's floor function) are garbled by PDF extraction; proofs were assessed at a conceptual level.
- Judge prompts (Appendix E) were reviewed textually, but judge reliability and inter-run variance of the GPT-5-mini analyses cannot be assessed from text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.