Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers
cs.LG · cs.AI
Quick verdict
LWD reweights teacher-failed trajectories in offline OPD via a rollout-free learnability signal; consistent but modest gains, limited statistical rigor.
Summary
The paper addresses imperfect teacher supervision in offline on-policy distillation (OPD), where student trajectories and frozen teacher targets are collected once and reused, so flawed supervision persists across optimization. The authors argue that a teacher's failure on a problem is a coarse label that does not condemn every token-level target on the student trajectory. Their method, Learnability-Weighted Distillation (LWD), first trains a reference student only on teacher-successful problems (successful-only learning), then scores stored tokens of teacher-failed trajectories by the change in token log-likelihood between the reference and the initial checkpoint. These smoothed signals are standardized within the failed subset, mapped to low/mid/high tiers, and turned into bounded trajectory weights (0.15/0.25/0.35, calibrated to a mean of 0.25) that scale the original reverse-KL OPD loss; successful records keep weight 1. The signal requires only two forward passes, no extra rollouts. Distilling Qwen3-8B into Qwen3-4B/1.7B students on DAPO-Math-17K and EpiCoder, LWD improves the protocol-matched Lightning-OPD baseline on all five benchmarks (e.g., AIME24 66.4% to 69.1% at 4B) and matches or beats online selective OPD on most math evaluations, at 2 GPUs and about 22 GPU hours. Limitations: no variance or significance reporting, code success labels are format-validity proxies, and the gain over uniformly downweighting failed cases is modest (+0.7 average).
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In offline OPD, a teacher's failure on a problem is a coarse signal that warrants graded, not binary, retention of the associated supervision.
- Signed token-likelihood changes induced by successful-only reference training form a cheap, rollout-free learnability signal computable once from stored trajectories and checkpoints.
- Aggregating this signal into tiered trajectory-level weights improves a protocol-matched Lightning-OPD baseline on all five benchmarks at two student scales, up to +2.7 points on AIME24 at 4B.
- Trajectory-level tiered weighting outperforms token-, span-, and continuous-trajectory weighting in the ablations.
- LWD preserves offline economics (2 GPUs, about 22 GPU hours), roughly half the GPU-hour cost of the compared online selective OPD methods.
Strengths
Well-motivated problem decomposition
Section 1 and Figure 1 clearly separate teacher outcome (problem-level) from per-token supervision value in offline OPD; the central question — how much, not whether, to retain failed-case supervision — is precise and underexplored relative to binary filtering (Zelikman et al. 2022; Yuan et al. 2023).
Cheap, rollout-free signal with an exact identity
Eq. 13 shows the mean token signal equals the trajectory log-likelihood ratio under the reference policy vs the initial policy, computable with two forward passes over stored trajectories; the contrast with prefix-continuation recoverability estimation (Fig. 1b) is convincing.
Ablations isolate the design choices
Consistent, protocol-matched empirical gains
Transparent cost accounting and implementation detail
Weaknesses
No variance or significance analysis
Thin margin of the core contribution over a simpler baseline
Weak success proxy for code
Hyperparameter-heavy pipeline without sensitivity analysis
Conditional superiority claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Learnability-Weighted Distillation (LWD)?
LWD is a method for offline on-policy distillation of language models. When the teacher fails on a problem, instead of keeping or discarding that trajectory outright, LWD first trains a reference student only on teacher-successful problems, then measures how much the likelihood of each stored token in teacher-failed trajectories changes. These changes are aggregated into low/mid/high trajectory-level weights that scale the original distillation loss, requiring only two forward passes and no new rollouts.
How much better does LWD perform than the offline baseline?
Distilling Qwen3-8B into Qwen3-4B-Base, LWD improves the protocol-matched Lightning-OPD baseline on all five benchmarks: AIME24 from 66.4% to 69.1%, AIME25 from 58.0% to 60.0%, HMMT25 from 40.5% to 42.1%, LiveCodeBench v5 from 42.6% to 44.5%, and v6 from 40.4% to 42.1%. Similar trends hold at the 1.7B scale, though online methods remain slightly stronger there on code.
How much compute does LWD need compared with online on-policy distillation?
The paper reports LWD uses 2 GPUs, about 11 wall-clock hours, and 21.9 GPU hours, versus 3 GPUs and roughly 36-48 GPU hours for online selective OPD baselines such as ReNIO and FiRe-OPD. Relative to the offline Lightning-OPD baseline, the extra successful-only distillation stage roughly doubles the cost, from 10.6 to 21.9 GPU hours.
Does the method need extra teacher rollouts to score trajectories?
No. Unlike continuation-based recoverability estimates, which sample many new continuations from trajectory prefixes, LWD scores the stored tokens with two forward passes (under the initial checkpoint and the successful-only reference checkpoint), so it can be computed once from stored trajectories and model checkpoints.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are extracted as fragmented text; the qualitative case study (Fig. 3) and weight illustrations could not be verified visually.
- Content on page 13, between Appendix A.3 and A.4, appears missing from the extraction; some implementation or figure material may be absent.
- The abstract in the supplied metadata is truncated mid-sentence; the full abstract is present in the body text and was used instead.
- No code or data release statement appears in the extracted text; codeAvailable is set to false based on absence, not an explicit statement.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.