What Does Privileged Information Add to On-Policy Self-Distillation?
XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
cs.CL
Quick verdict
OPSD gains mostly come from cross-mode distillation, not the reference; rigorous controlled study, though key effects are small and several runs use one seed.
Summary
The paper asks how much a privileged reference (answer or worked solution) adds to on-policy self-distillation (OPSD), where a frozen, thinking-enabled copy of a model scores a trainable student's own rollouts. The authors build AMPLE-Math, 5,319 math problems from OpenThoughts-114k, each with six answer-matched reasoning views (Answer Only to Full Trace, 13 to 4,916 mean tokens), and compare each against a matched reference-free control in Qwen3-1.7B and SmolLM3-3B using LoRA over 100 steps. Under direct-response training with thinking-enabled evaluation, reference-free distillation accounts for most of Qwen's in-domain and external gains; only Clean Solution shows a small edge (+1.30 points at step 100) that fails Holm multiple-comparison correction. In SmolLM3, Full Trace adds about 2 points at step 50, yet all configurations fall below the base by step 100. Replacing short direct-response training rollouts with long thinking-enabled ones reverses gains into losses in both families under an inherited 1,024-token loss window. Teacher profiles show correction-marker suppression in both modes, but relaxing marker loss changes student behavior little, and the apparent benefit of a broader loss window reduces to stopping earlier at matched checkpoints. Limitations: short LoRA runs, mathematics only, two small models, many single-seed comparisons, and base models already near ceiling.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reference-free cross-mode distillation accounts for most of Qwen3-1.7B's OPSD improvement under thinking-enabled evaluation, in domain and externally.
- The additional benefit of a privileged reference is modest and view-specific: only Clean Solution shows a small unadjusted advantage that fails Holm correction, while SmolLM3's Full Trace adds about 2 points at step 50.
- With teacher and references fixed, replacing short direct-response training rollouts with long thinking-enabled rollouts turns gains into losses in both model families.
- Changing token-level supervision (marker-loss relaxation, broader loss windows) leaves student behavior largely unchanged, and the apparent loss-window gain is explained by stopping earlier at matched checkpoints.
Strengths
Genuinely controlled experimental design
AMPLE-Math holds the verified answer fixed across six views (13–4,916 mean Qwen3 tokens) per problem, and the reference-free control preserves the cross-mode asymmetry (thinking teacher scoring direct-response prefixes), so contrasts isolate the reference rather than distillation itself; a wrong-answer control (Table 1: +1.67 over base) further probes content relevance.
Statistical discipline above field norms
Paired problem-cluster bootstrap (10,000 resamples, shared resample arrays), stratified resampling, Holm correction over explicitly declared families, and separate seed-level t/Welch checks (Appendix A.4, Tables 9–10); the authors volunteer that their only positive Qwen contrast (Clean Solution, raw p = 0.024) has Holm-adjusted p = 0.15.
Checkpoint-matched negative result on loss design
Cross-family replication with decontamination
Reproducibility scaffolding
Weaknesses
Headline positive effect is statistically fragile
Heavy reliance on single training seeds
Small margins against near-ceiling baselines
Confound in the trajectory-reversal experiment
Intricate grading protocol with untested sensitivity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: in on-policy self-distillation (OPSD), where a frozen thinking-enabled teacher scores a trainable student's own rollouts, how much does giving the teacher a privileged reference (answer or worked solution) add beyond distillation itself? The central claims are: (1) reference-free cross-mode distillation accounts for most of Qwen3-1.7B's improvement (+1.80 in-domain at step 100, Table 1); (2) the reference's marginal value is modest and view-specific — only Clean Solution beats the reference-free control with an unadjusted +1.30 [0.20, 2.41] that fails Holm correction; (3) in SmolLM3-3B, Full Trace adds +2.00 [0.82, 3.17] at step 50, yet all configurations fall below the base by step 100 (Table 6); (4) swapping direct-response rollouts for thinking-enabled ones reverses gains into losses in both families (all nine Qwen gaps negative; pooled −7.60 at step 50, Table 11); and (5) supervision edits barely change student behavior, with the loss-window gain reducing to stopping time. All claims are appropriately scoped to short-run LoRA training on mathematics.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AMPLE-Math?
A released suite of 5,319 math problems (built from OpenThoughts-114k) in which every problem comes with six reasoning 'views' — answer only, gist, key points, clean solution, summary, and full trace — that all share the same verified answer. It is designed to test what the content of a teacher-side reference contributes to on-policy self-distillation while holding solution information fixed.
Does giving the distillation teacher the full solution help the student model?
According to this paper, mostly not: in Qwen3-1.7B most of the improvement appears even with no reference at all, and the only positive view contrast (+1.30 points for polished solutions) does not survive multiple-comparison correction. In SmolLM3-3B, full traces add about 2 points at step 50, but by step 100 every trained configuration scores below the untrained base model.
What is on-policy self-distillation (OPSD)?
A training method in which a frozen copy of the same language model — optionally given privileged information such as the answer or a worked solution — scores the responses that the trainable student model generates itself, typically via a KL-divergence loss on the student's own tokens. The student thus learns from teacher feedback on states it actually visits, without needing a larger separate teacher.
Is the code and dataset for this paper available?
Yes. The paper's first page links to code on GitHub and the AMPLE-Math dataset on Hugging Face. The appendices document full hyperparameters, prompt templates, statistical procedures, and dataset construction, and the authors state row-level evaluation outputs are available on request; reproducing training requires roughly 1,000 H100 GPU-hours of compute.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted captions and axis labels; visual content (e.g., Figure 2 curves, Figure 4 profiles) could not be independently verified against the tables.
- The extracted text ends at Table 18 with no visible truncation, but minor formatting of appendix tables may be garbled by extraction.
- Author lists differ slightly between the metadata (five authors) and the paper header (six, including Xingyu Zhu); the review uses the paper header.
- Many cited works are 2026 arXiv preprints; claims about how this study relates to concurrent work were not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.