From Base Rollouts to RL Reasoning: A Budgeted Search Perspective
Wenhe Sun, Cunxiang Wang, Zijun Yao, Yixin Cao
cs.CL
Quick verdict
Argues RLVR gains are largely recoverable by budgeted base-model search (BOPTR); rigorous, honest study, but edge over greedy baseline is within noise.
Summary
The paper asks whether RL with verifiable rewards (RLVR) creates reasoning the base model lacks or merely improves sampling efficiency. Using SearchLens, a Unified Decoding Framework expressing sampling, beam-like, tree, and resampling policies as operating points (pi, N) over budgets N in {1,2,4,8,16}, scored post hoc with pass@k, self-consistency, best-of-N and first-finish, the authors test whether SimpleRL-Zoo RL default-policy curves can be recovered by structured Base+search paths. The recovery budget follows BOPTR, N_Base ≈ alpha·N_RL^beta, with regime-conditioned exponents (math 0.6, non-math 1.0, floor 0). On Qwen2.5-7B the rule attains 3.41 pp mean transfer error (95% CI [2.32, 5.53]; three-seed replication 3.07±0.39), extending to ten models across four families (3.28–4.87 pp), four unfitted benchmarks (5.03 pp), and settings with no target-RL data (4.19 pp) or no RL at all (5.08 pp). Limitations: the regime taxonomy derives from only four fitted benchmarks and the floor exponent is statistically indistinguishable from alternatives; AIME has n=60; GPQA cross-batch readout drift (±3–4 pp) approaches several reported cell errors; and the rule's improvement over plain greedy decoding is at parity (paired difference −0.04 pp). Claims are carefully scoped as behavioral diagnostics, not parameter-level equivalence.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RL default-policy pass@k curves under the SimpleRL-Zoo recipe are recoverable by structured Base+search operating paths whose budget follows N_Base ≈ α·N_RL^β with regime-conditioned exponents (math 0.6, non-math 1.0, floor 0).
- On the Qwen2.5-7B anchor, BOPTR-P1 achieves the lowest transfer error among tested non-oracle rules: 3.41 pp (95% CI [2.32, 5.53]; three-seed 3.07±0.39).
- The frozen rule transfers to ten models across four families (3.28–4.87 pp), four unfitted benchmarks (5.03 pp), and settings with no target-RL checkpoint (4.19 pp) or no RL supervision at all (5.08 pp).
- Much of the measured RL gain corresponds to improved sampling efficiency toward operating points the base model can already reach under search — a behavioral claim, explicitly not parameter-level equivalence.
- The self-consistency channel marks a genuine limit: 19/23 cells are recoverable, but four Math500 cells (Qwen 0.5B/1.5B/14B, Llama) show RL SC sharper than any base policy at any budget (Table 14).
Strengths
Unusually rigorous uncertainty quantification
Question-level paired bootstrap CIs (B=10^4) on every Table 17B row, paired-difference CIs vs baselines, a 3-seed decoding replication (3.07±0.39 pp), Wilson half-widths quoted for scale, and an honest negative result: the V4 base-only ridge sits inside the y-shuffle null (LOO pseudo-R² −0.38, p≈0.21, Table 15).
Disciplined claim scoping and negative-result reporting
The floor regime β=0 is kept only as a descriptive label after showing the two-regime counterfactual is indistinguishable (Table 19A: Δ +1.67, CI [−5.00, +4.00]); H1 parity is disclosed; v6/v7 rule variants are reported as negative ablations (A.6.1); a robustness checklist maps each headline claim to its guardrail (A.6.2).
Recovery methodology controls post-hoc matching
Broad frozen-parameter transfer coverage
Weaknesses
Headline advantage over the simplest baseline is within noise
Regime taxonomy rests on four benchmarks and five budgets
Small, noisy evaluation cells and a cross-batch caliber problem
Inconsistent cell inventories across tables
Coarse budget grid limits the fitted power law
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tackles a contested post-training question: does RLVR create reasoning the base model lacks, or shift the rollout distribution toward trajectories the base can already reach? The answer is behavioral. SearchLens (UDF) expresses token sampling, beam-like, tree, and resampling policies as operating points (π, N) over budgets N∈{1,2,4,8,16}; pass@k, self-consistency (SC), best-of-N, and first-finish are applied only after generation. Using paired Base/RL checkpoints from SimpleRL-Zoo, the central claim is that RL default-policy pass@k curves are recoverable by low-complexity Base+search paths obeying BOPTR, N_Base ≈ α·N_RL^β, with regime exponents (math 0.6, non-math 1.0, floor 0). Claims are explicitly qualified: behavioral, not parameter-level; recipe- and cohort-bound (Limitations section; A.6.2 robustness checklist).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does this paper prove that RL training teaches language models nothing new?
No. It argues behaviorally that, for the SimpleRL-Zoo recipe tested, many RL pass@k gains can be matched by the base model using suitable decoding/search at adjusted budgets. The authors explicitly disclaim parameter-level equivalence, and their own self-consistency analysis finds four Math500 cells where the RL model is sharper than any base policy at any budget.
What is BOPTR in this paper?
BOPTR (Budgeted Operating-Point Transition Rule) is a one-line rule predicting which base-model sampling budget reproduces an RL model's pass@k: N_Base ≈ α·N_RL^β, with β fixed per benchmark regime (0.6 for math, 1.0 for non-math/OOD, 0 for the AIME 'floor' regime). It reached about 3.4 percentage points mean error on Qwen2.5-7B and transferred to other models and benchmarks without refitting.
Is code and data available for this paper?
The paper states that code, configurations, and analysis scripts are released at github.com/HALIS-sh/Searchlens_boptr; models are public SimpleRL-Zoo checkpoints and standard public benchmarks. Note the full anchor-cell sweep cost about 461 H100 GPU-hours, mostly from eight expanded search-controller runs, so full replication is compute-heavy even though the AR-only rule components are far cheaper.
What are the main limitations of the study?
The regime taxonomy is derived from only four fitted benchmarks and the floor exponent is statistically indistinguishable from alternatives; evaluation sets are small (AIME n=60, GPQA n=198); GPQA readouts drift ±3–4 pp across sampling batches; and on the anchor benchmark set the rule shows no statistically significant accuracy gain over plain greedy decoding — its value lies in structure and cross-model transfer.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as captions in the extraction; curve-shape judgments rely on reported numbers and caption text rather than visual inspection of the figure panels.
- codeAvailable and reproReadiness reflect the paper's artifact claims; the GitHub repository itself could not be inspected (no network access in this review).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.