RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, et al.
cs.CL · cs.AI
Quick verdict
Proposes online teacher retirement for GRPO+OPD agents; +12–19 pts on ALFWorld/WebShop over GRPO; strong tables but single-run, no variance reported.
Summary
The paper identifies two failure modes of privileged-information on-policy distillation (OPD) for LLM agents: a skill-conditioned teacher that has not itself been RL-trained is unreliable (a 7B model given skills in context reaches only 23.4% on ALFWorld), and teacher supervision becomes counterproductive once the student internalizes the skills, because GRPO and distillation gradients begin to conflict — the teacher–student gap narrows, then rebounds (a first-order argument appears in Appendix A). RetireOPD therefore first trains the skill-conditioned teacher with GRPO, then trains a skill-free student jointly with GRPO and reverse-KL OPD on student rollouts, monitoring window-averaged log-probability gaps and relative success rate; when gap reduction stalls and the student reaches 90% of teacher success, OPD is retired and training continues with GRPO alone. Across Qwen2.5-1.5B/3B/7B on ALFWorld and WebShop, RetireOPD improves over GRPO by 14.1–18.8 points (ALFWorld) and 11.8–19.0 points (WebShop accuracy), surpasses its own teacher in all six settings, and beats GiGPO, GRPO+OPD, and linear annealing. Main limitations: single runs without variance reporting, and the extra teacher-training compute is never accounted for.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A skill-conditioned teacher is reliable only after being optimized with environment rewards: privileged context alone yields poor teachers (unoptimized 7B: 23.4% on ALFWorld; after GRPO: 90.6%).
- Teacher supervision is stage-dependent: once the teacher–student discrepancy stops shrinking, OPD conflicts with GRPO and caps the student near the teacher's performance.
- Online retirement triggered by discrepancy stagnation plus a 0.9 relative-competence threshold outperforms fixed schedules (linear annealing), continuous joint training, and either criterion alone.
- RetireOPD improves ALFWorld success by 14.1–18.8 points and WebShop accuracy by 11.8–19.0 points over GRPO across Qwen2.5-1.5B/3B/7B, and surpasses its own skill-conditioned teacher in every setting.
Strengths
Empirically grounded motivation
Figure 2 and §4.3 document both failure modes directly: an unoptimized 7B with skills in context reaches only 23.4% on ALFWorld (below GRPO's 81.2, Table 1), and the teacher–student gap narrows then rebounds; the continue-from-retirement-point experiment (Fig. 4) cleanly isolates the conflict — joint training plateaus, OPD-only degrades, GRPO-only climbs 76.6→93.8.
Simple dual-signal retirement criterion, properly ablated
Table 2 shows both signals contribute (82.8 with no retirement → 89.0 with either criterion alone → 92.2 with both); Table 3 and Fig. 7 show the retiring step moves only ±5 steps over γ∈[0.80,0.96] and δ∈[−0.10,0.04] with success 89.1–92.2 — a genuine robustness result.
Comprehensive unified comparison
Student surpasses its own teacher in all six settings
Honest theoretical scoping
Weaknesses
No variance, seeds, or significance testing
Internal numeric inconsistency
Compute accounting absent
Unanalyzed baseline collapse
Narrow generality evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RetireOPD (arXiv 2609.20784, Zhejiang University + Alibaba) is a method paper for training multi-turn LLM agents. It targets the now-standard recipe of privileged on-policy distillation (OPD), where a student reproduces a same-capacity teacher conditioned on privileged skills available only during training. Two diagnostic claims, supported by Figure 2 and §4.3: (1) privileged context alone does not make a reliable teacher — an unoptimized 7B with skills in context reaches only 23.4% on ALFWorld, below GRPO's 81.2 (Table 1); (2) teacher supervision is stage-dependent: the teacher–student log-probability gap narrows, then rebounds once reward optimization favors actions the teacher disfavors. The constructive claim is that online teacher retirement, triggered by gap stagnation (ρ ≥ 0) plus relative competence (η ≥ 0.9), beats fixed schedules and continuous joint training, with +14.1 to +18.8 points over GRPO on ALFWorld and +11.8 to +19.0 on WebShop accuracy across Qwen2.5-1.5B/3B/7B.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RetireOPD in one sentence?
RetireOPD is a training recipe for LLM agents that first trains a skill-conditioned teacher with reinforcement learning, then trains a skill-free student with joint RL (GRPO) and on-policy distillation, and automatically drops ('retires') the teacher once the student stops getting closer to it and reaches 90% of the teacher's success rate.
How much better is RetireOPD than standard GRPO RL?
On ALFWorld, success rate improves by 14.1–18.8 points over GRPO across Qwen2.5-1.5B/3B/7B (e.g., 93.8% vs 75.0% at 3B); on WebShop, accuracy improves by 11.8–19.0 points (e.g., 75.8% vs 56.8% at 1.5B). The trained student also surpasses its own teacher in all six reported settings.
When does the teacher get retired during training?
Every five training steps the method checks two signals: whether the window-averaged teacher–student log-probability gap has stopped shrinking, and whether the student's success rate has reached 90% of the teacher's. When both hold, distillation is switched off and training continues with GRPO alone.
Is code or data available for RetireOPD?
The paper states code is available at github.com/ZJU-REAL/SDAR.OPD. Experiments use public ALFWorld and WebShop environments, SkillRL's SkillBank as the skill source, and Qwen2.5-1.5B/3B/7B-Instruct models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 4–7 and 10–13 are present only as axis labels and embedded numbers; curve shapes and per-window dynamics could not be visually verified.
- arXiv v1 preprint (17 Sep 2026), not yet peer-reviewed.
- Source metadata lists 8 authors while the extracted text shows 11; the author list in the metadata may be truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.