RL Starts before RL: On Policy Distillation for Better Reinforcement Learning
Shuai Dong, Yongfu Zhu, Yuqi Xu, Weichu Xie, Liuwenpu, Ziyue Wang, Kaiwen Tuo, Congcong Wang, et al.
cs.LG
Quick verdict
Claims OPD preparation beats Base/SFT starts for RL and the RKL/FKL ranking flips post-RL; strong diagnostics but single-seed runs limit small margins.
Summary
The paper asks whether on-policy distillation (OPD) prepares language models for better reinforcement learning beyond raising their initial accuracy. Using Qwen3-8B/Qwen3-32B (text) and Qwen3-VL-8B/32B (vision-language) pairs, the authors compare direct RL, SFT→RL, and OPD→RL under shared GRPO-style settings. OPD initialization reaches the highest final averages — 59.18 versus 51.90 (Base) and 47.48 (SFT) on eight text benchmarks, and 63.02 versus 61.68 and 61.33 on twenty VL benchmarks — often from nearly unchanged pre-RL accuracy. Pre-RL Pass@k does not explain the gap: text Base and OPD share nearly identical Pass@k curves, and RKL/FKL variants with identical Pass@64 (92.68%) still diverge after RL. Diagnostic probes show OPD lowers Jensen–Shannon divergence from the teacher (0.01468 versus SFT's 0.01720 in text) while retaining token entropy (1.048 versus 0.242), supporting a hypothesis that distributional alignment plus retained uncertainty benefits subsequent RL. In a VL-only recipe study, forward-KL OPD overtakes reverse-KL OPD after RL (64.87 versus 63.02) despite equal coverage, while reverse KL stays ahead with teacher-generated trajectories. Key limitations: one seed per pipeline with no cross-run variance for RL margins, an unspecified checkpoint-selection criterion, no compute-matched comparison, and a correlational mechanism.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- OPD-initialized RL reaches the highest final benchmark averages in both settings (text 59.18 vs 51.90 Base and 47.48 SFT; VL 63.02 vs 61.68 and 61.33; Table 1).
- Pre-RL Pass@k does not fully predict post-RL performance: text Base and OPD have near-identical curves (Pass@64 80.37 vs 79.98), and RKL/FKL variants share Pass@64 of 92.68% yet differ by 1.85 points after RL.
- OPD lowers teacher JS divergence below both Base and SFT while retaining far higher token entropy than SFT (text 1.048 vs 0.242 nats), proposed as the mechanism behind its downstream benefit.
- The preferred KL objective depends on trajectory source: FKL overtakes RKL after RL on student trajectories (64.87 vs 63.02), while RKL leads at both stages on teacher trajectories (61.90 vs 59.20).
Strengths
Well-posed, practically relevant question with a controlled design
Within each setting, Base/SFT/OPD checkpoints come from the same student and feed identical GRPO-style RL (data, verifier, optimizer, sampling; Sec. 3.2, App. A), isolating the effect of the starting policy; the 2×2 recipe study (trajectory source × KL objective, Table 14) exposes a ranking reversal a single-stage evaluation would miss.
Diagnostic battery goes beyond benchmark scores
Finite-sample Pass@k and group-outcome estimators (Eq. 3–4) over 1,024 shared prompts with paired bootstrap intervals (Table 11), top-20 JS teacher-alignment probes at each policy's own prefixes (Table 12), and lexicon-conditioned entropy split by response correctness (Table 13) — explicitly showing top-1 agreement alone is misleading (SFT 94.25% vs OPD 93.82% top-1, but higher JS, in text).
Honest self-disclosure and complete reporting
The negative result on Pass@k is itself useful
Weaknesses
Single run per pipeline; no seed variance for the headline margins
Unspecified checkpoint selection invites selection bias
No compute-matched comparison
Correlational mechanism and narrow scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a training-pipeline question: does on-policy distillation (OPD) — the student sampling its own responses while matching a stronger teacher's full next-token distributions — prepare a model for reinforcement learning better than supervised fine-tuning (SFT) or no preparation, even when it barely improves accuracy? Four claims are advanced: (1) OPD→RL yields the highest final benchmark averages in both settings (text: 59.18 vs 51.90 Base→RL and 47.48 SFT→RL; VL: 63.02 vs 61.68 and 61.33; Table 1); (2) pre-RL Pass@k does not explain this (text Base/OPD curves nearly coincide, Table 8; the RKL/FKL pair with identical Pass@64 = 92.68% still separates after RL, Sec. 6); (3) OPD combines lower Jensen–Shannon divergence from the teacher with retained token entropy (Tables 12–13), proposed as the mechanism; (4) the preferred KL objective interacts with trajectory source: FKL overtakes RKL after RL on student trajectories (64.87 vs 63.02), while RKL leads throughout on teacher trajectories (61.90 vs 59.20; Table 14).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the title 'RL Starts before RL' mean?
The paper argues that what happens before reinforcement learning matters: initializing RL from an on-policy distilled model led to higher final scores than starting from the base model or an SFT model, even when the distillation stage barely improved accuracy beforehand.
What is on-policy distillation (OPD)?
A training method where the student model generates its own responses and a stronger teacher provides full next-token probability distributions at each position of those responses for the student to imitate (typically via reverse KL divergence), unlike SFT, which learns from fixed teacher-written text.
Should I use forward KL or reverse KL for on-policy distillation before RL?
In this paper's vision-language experiments, reverse-KL students scored higher right after distillation, but forward-KL students scored higher after subsequent RL (64.87 vs 63.02 across 20 benchmarks). With teacher-generated trajectories the ordering stayed with reverse KL. The authors advise choosing the objective by post-RL performance, not distillation-time scores.
Is code available for this paper?
No code or data release is mentioned in the paper text. However, the ingredients are public: Qwen3/Qwen3-VL models, OpenMathInstruct, DAPO-Math, OpenMMReasoner and related datasets, plus hyperparameters in Appendix A (Table 4).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as embedded numeric values in the extracted text; curve shapes (e.g., Fig. 3 Pass@k curves) were judged from the corresponding tables.
- The metadata abstract is truncated mid-sentence, but the complete abstract appears in the body text.
- No code/data availability statement was found in the extracted text; its absence could be an extraction artifact rather than a fact about the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.