When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
Yuxiao Yang, Tianrun Yu, Shangzhe Li, Kaixiang Zhao, Xuchao Zhang, Xuchao Zhang, Chetan Bansal, Chetan Bansal, et al.
cs.LG
Quick verdict
Shows student–teacher EOS-token mismatch drives OPD length inflation; probability-level alignment mitigates it across 3 model families. Plausible, small-scale.
Summary
On-policy distillation (OPD) trains a student on its own rollouts scored by a fixed teacher, yet student responses often inflate until they exhaust the generation budget. This paper identifies a concrete mechanism: termination-token mismatch. Across Qwen3, Llama 3.2, and Gemma 3 base-to-post-trained pairs, student and teacher place their stopping probability on different EOS tokens, even when declared stopping sets are identical (Gemma 3's PT student favors <eos>, its IT teacher <end_of_turn>). Because sampled-token OPD scores only the token the student actually samples, the student's native EOS receives repeated negative signal, while the teacher-preferred token — with roughly 10^-11 student probability in Qwen3 — is almost never sampled. On DAPO-Math-17K, distilling Qwen3-4B into Qwen3-1.7B-Base shows that aligning the decoding stopping set alone fails, whereas three probability-level corrections (teacher-side EOS mapping, semantic EOS aggregation, canonical single-EOS action space) keep length and clipping near teacher levels. A stage-wise K2-Horizon analysis shows termination preferences shift across training stages, and late-stage re-inflation (Phase 3) persists under semantic correction, indicating an unexplained second mechanism; Proposition 1 in Appendix C offers omitted successor-KL stopping pressure as a candidate explanation. Limitations: single-turn math only, small models, modest model-dependent accuracy gains, tiny benchmarks, single-seed runs.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Base students and post-trained teachers across Qwen3, Llama 3.2, and Gemma 3 place their learned stopping probability on different EOS tokens, even when their declared stopping sets are identical.
- In sampled-token OPD, this mismatch repeatedly suppresses the student's native EOS while the teacher-preferred token (student probability ~10^-11 in Qwen3) is almost never sampled, driving length inflation and budget exhaustion.
- Aligning the decoding stopping set alone does not fix inflation, whereas probability-level corrections — teacher-side EOS mapping, semantic EOS aggregation, and canonical single-EOS action space — substantially mitigate it across all three families.
- Termination preferences shift substantially across K2-Horizon training stages, and a late-stage length re-inflation (Phase 3) persists even after termination alignment, indicating a second, unexplained mechanism.
- Local sampled-token OPD applies weakly less upward pressure to the stopping logit than trajectory-consistent reverse KL, by a nonnegative gap pθ(s)Cθ(s) (Proposition 1), plausibly explaining transient early length growth after correction.
Strengths
Fix ladder cleanly isolates the mechanism
On Qwen3, the four-way ablation (Figure 4a) shows decoding-only shared-set decoding (Fix 1) tracks vanilla OPD's inflation, while three probability-level corrections (Fixes 2–4) stay near teacher references — the experiment that separates decoding interface from distillation signal.
Generalizes beyond configuration mismatch
Gemma 3 (Figure 3) exhibits the failure even though PT and IT checkpoints declare identical stopping sets (<eos> vs <end_of_turn> learned preferences), showing the mismatch is about learned probability mass, not just generation_config declarations; replicated on Llama 3.2 and Qwen3.
Non-interference control for the fix
Mechanistic, quantitative diagnosis
Honest demarcation of what is explained
Weaknesses
No variance quantification anywhere
Downstream payoff is thin
Key counterfactual never run
Residual mechanism unexplained
Partial novelty overlap
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.20511, cs.LG) studies length inflation in on-policy distillation (OPD): students trained on their own rollouts under token-level teacher scores grow responses until the generation budget is exhausted. The authors claim a concrete, diagnosable cause — termination-token mismatch — present across Qwen3, Llama 3.2, and Gemma 3 base-to-post-trained pairs, including cases where declared stopping sets are identical (Gemma 3's PT student favors <eos>, its IT teacher <end_of_turn>; Figure 3). Three central claims: (1) decoding-level stopping-set alignment (Fix 1) is insufficient; (2) probability-level corrections — teacher-side EOS mapping (Fix 2), semantic EOS aggregation (Fix 3), canonical single-EOS action space (Fix 4) — mitigate inflation, validated across three families; (3) a K2-Horizon stage-wise study shows termination preferences drift across training stages, and late-run re-inflation (Phase 3, Figure 7) persists after alignment, so EOS mismatch is 'important, but not exhaustive.' Proposition 1 (Appendix C) shows local sampled-token OPD applies weakly less stopping-logit pressure than trajectory-level reverse KL, by a nonnegative gap pθ(s)Cθ(s).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do on-policy distillation (OPD) students produce endless repetitive outputs?
This paper argues one major cause is EOS-token mismatch: the base student and the post-trained teacher place their stopping probability on different special tokens (e.g., Qwen3's base model uses <|endoftext|> while the post-trained model favors <|im_end|>). Since OPD only scores the token the student actually sampled, the student's own stop token keeps receiving negative signal and the student effectively stops learning to terminate, so responses inflate until the generation budget is exhausted.
Does the proposed EOS alignment fix improve accuracy on math benchmarks?
Mostly no. The fix mainly shortens responses and prevents budget exhaustion. The paper reports modest, model-dependent accuracy changes: on Llama 3.2, accuracy stayed near zero both with and without the fix, while Qwen3 and Gemma 3 show small changes on AIME24/25 and AMC23 (Avg@16 over 16 samples per problem).
Do the student and teacher need different declared EOS configurations for this problem to occur?
No. Gemma 3 is the key counterexample: its PT and IT checkpoints declare the same stopping tokens (<eos> and <end_of_turn>), yet the student places its stopping probability on <eos> while the teacher prefers <end_of_turn>. A learned-probability mismatch within a shared declared set is enough to trigger the failure.
Is length inflation in on-policy distillation now solved?
No. The termination-token correction removes one identified cause, but the K2-Horizon experiments show a late-stage re-inflation (Phase 3) that persists even under semantic EOS correction, and the authors explicitly leave that second mechanism unexplained. Prior works also report inflation in settings without any declared stopping-set mismatch.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–13 carry most quantitative results; only the text's verbal descriptions of them were available, so curve shapes and magnitudes could not be independently verified.
- Three spans were omitted in extraction (part of Section 4.1, part of Appendix B.3, and Appendix E Observation 2), so some details may be missing.
- The code release and the DAPO-grader extension are stated but could not be verified from the extracted text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.