RISE: Recursive Improvement via Self-Extrapolating Policy Distillation
Yang Li, Semih Yavuz, Shafiq Joty
cs.AI
Quick verdict
Proposes self-extrapolating teacher from own RLVR trajectory; consistent gains over GRPO/OPSD at 1.7-8B with honest ablations, though gains shrink with scale.
Summary
The paper introduces RISE, a post-training method that constructs a synthetic teacher from the model's own RLVR training trajectory rather than from an external model or privileged conditioning. After each GRPO update, RISE extrapolates the displacement between the current checkpoint and a trailing anchor (β>1, decayed from 1.2 to 1) in parameter space (task arithmetic) or logit space (geometric mixture), then distills the extrapolated teacher into the student with a top-K Jensen–Shannon loss, reusing RLVR rollouts at no extra sampling cost. Low-rank, near-linear trajectory evidence (top-3 components ≈87% of variance) and a suboptimality decomposition motivate the design; the theory stays qualitative, and reverse KL is replaced by JSD in practice. RISE beats GRPO in every tested setting — mathematical reasoning (Qwen3-1.7B/8B, OLMo3-7B), mixed math+STEM (Qwen3-4B-Base), code generation, and agentic tasks (ALFWorld/WebShop) — and beats the three privileged-teacher OPSD baselines in the math and multi-domain settings: OLMo3-7B AIME'24 +16.7 points, Qwen3-1.7B Math Avg +4.8, ALFWorld +9.4. Gains are seed-robust (three seeds, std ≤0.6) and hold against a compute-matched GRPO-2× control. Ablations show both phases are needed: without RLVR grounding, training collapses within 60 steps; without OPD, gains vanish. Main limitations: gains shrink with scale (8B: +2.7), anchor hyperparameters differ per model family (chosen post hoc), code results lack numeric tables, no significance tests are reported, and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A synthetic teacher built by extrapolating the model's own RLVR displacement (weight space via task arithmetic, or logit space via geometric mixture, β>1) provides effective dense per-token supervision without external models or privileged conditioning.
- RISE outperforms GRPO in every tested setting and the three privileged-teacher OPSD baselines in math and multi-domain settings, at 1.3–1.6× GRPO wall time with no additional sampling cost.
- The safe extrapolation range narrows as training converges (Proposition 3, Fig. 6), empirically justifying the monotonically decaying β schedule.
- Both phases are necessary: RLVR grounding prevents self-referential collapse, and OPD — not direct adoption of the extrapolated weights — produces the gains.
Strengths
Teacher-construction principle is measured, not just asserted
Appendix A.3 quantifies the linear-trajectory premise on their own runs: top-3 principal components capture 86.6%/88.5% of displacement variance at 1.7B/8B; Fig. 6 directly measures the narrowing safe-β range (β=2.0: +7.3 pts at step 50, −15 pts at step 100 on 1.7B AIME'24), empirically justifying the decaying β schedule rather than merely citing prior low-rank work.
The right control experiments are present
Compute-matched GRPO-2× (Table C.7) shows RISE's edge survives equal gradient budget (+2.9 at 1.7B, +2.2 at 8B); phase-removal ablations (Fig. 5, C.3) show w/o-RLVR collapses within 60 steps (MATH-500 → 2.4%, length → 8K cap) and w/o-OPD forfeits gains (60.0 → 60.3 at 8B); pass@16 vs avg@16 (Table C.2) rules out pure sharpening.
Broad, multi-seed-supported evaluation
Honest scoping and self-criticism
Weaknesses
Gains shrink with scale and evidence stops at 8B
Post-hoc per-family hyperparameter selection on evaluation benchmarks
Code-generation evidence is weak
Privileged baselines look under-tuned
Statistical reporting is minimal
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RISE (Salesforce AI Research; arXiv 2609.05295) targets the teacher-quality bottleneck in on-policy distillation (OPD). Its claim: a reliable teacher can be synthesized from the model's own RLVR trajectory by extrapolating the displacement between the post-RLVR checkpoint θ′n+1 and a trailing anchor θanchor with scale β>1 (Eq. 4), instantiated in weight space (task arithmetic, Eq. 5) or logit space (geometric mixture π^(1−β)·π′^β, Eq. 6). RLVR grounds the direction; OPD distills the extrapolated teacher per token; the teacher refreshes every iteration, making distillation recursive. The paper claims superiority over GRPO and three privileged-conditioning OPSD baselines (GRPO+SDPO, SDAR, RLSD) across math, mixed STEM, code, and agentic tasks at 1.7B–8B, at 1.3–1.6× wall time and no extra sampling. External-teacher OPD is explicitly out of scope (§2.4, B.3) — a scoping decision stated rather than hidden.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RISE and how does it differ from standard distillation?
RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a post-training method for LLMs. Instead of distilling from a bigger external model, or from the model itself given the correct answer, it builds a 'future teacher' by extrapolating the model's own training progress: it takes the parameter (or logit) change between the current checkpoint and an earlier anchor, amplifies it by a factor β>1, and distills that extrapolated model back token-by-token. The teacher is refreshed every iteration, so distillation becomes a recursive loop tied to the model's verified improvement.
Does RISE need a stronger teacher model or extra data?
No. The paper's core claim is that no external model, privileged solution, or extra data is needed — the teacher is synthesized from the model's own checkpoints along its RLVR trajectory. Training costs about 1.3–1.6× a GRPO run in wall time with no additional sampling, because the distillation phase reuses the reinforcement-learning rollouts.
How much does RISE improve over GRPO?
Reported gains shrink with model scale: on Qwen3-1.7B the math benchmark average rises from 45.4 to 50.2 (+4.8); on Qwen3-8B from 60.0 to 62.7 (+2.7, weight-space variant); and on OLMo3-7B AIME'24 jumps from 30.2 to 46.9 (+16.7). On the agentic benchmark ALFWorld the weight-space variant gains +9.4 success rate. Across three random seeds the math-average standard deviation is ≤0.6 for the replicated configurations.
What happens if you remove the RLVR part of RISE?
It collapses. The authors show that extrapolating along displacement caused by self-distillation alone (no verifiable-reward grounding) drives training into degenerate non-terminating generation within about 60 steps: MATH-500 accuracy falls to 2.4%, response length explodes to the 8K-token context cap, and training reward drops to zero. RLVR's outcome reward is what keeps the extrapolation direction pointed at genuine improvement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are extracted only as axis labels, titles, and legend text; no numeric curve values are readable, so every quantitative claim in this review rests on the in-text numbers and appendix tables (which are complete).
- The provided text ends cleanly after Table C.8 with the page-28 marker and complete references; no truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.