GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong
cs.AI · cs.LG
Quick verdict
GVPO++ extends GVPO to on-policy distillation: zero-sum weights drop importance sampling with exact-optimum theory; elegant, but 7B gains lack error bars.
Summary
This journal paper (extending a NeurIPS GVPO conference version) proposes GVPO++, a post-training method for LLMs that expresses training gradients as zero-sum weighted log-likelihood gradients, letting the partition function of the KL-regularized optimal policy cancel. The resulting loss equals a mean squared error between implicit-reward and actual-reward central distances, and the authors prove at population level that its unique optimum coincides with the KL-constrained reward-maximization solution for any sampling distribution covering the reference support — no importance sampling needed. The framework extends to sequence-level on-policy distillation (GVPD), including a family of weighted reverse-KL objectives where the teacher stays the exact optimum, enabling cross-tokenizer distillation; a length-normalizing weight f=1/|y|^α counters OPD length bias. On Qwen2.5-Math-7B trained on Competition Math, GVPO averages 52.02 across five math benchmarks versus 46.69 for GRPO and 49.31 for GSPO; on 1.5B, 10-seed runs give GVPO 15.56±1.05 vs GRPO 13.59±1.11 on AIME24; cross-vocabulary distillation from DeepSeek-R1-Distill-Llama-8B reaches 35.99 average versus 24.75 for policy gradient. Ablations show both implicit regularization terms (variance and covariance) are necessary for stability. Limitations: the theory is population-level with no finite-sample or convergence analysis; headline 7B results lack error bars, and seed variance is reported only against GRPO at 1.5B; the distillation weighting requires per-teacher α tuning; plot-based ablation claims cannot be verified from the text.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GVPO's gradient equals that of a mean squared error loss between group-centered implicit rewards (β·log πθ/πθ′) and actual rewards (Section IV-B).
- GVPO's unique global optimum coincides exactly with the KL-constrained reward-maximization policy for any sampling distribution whose support covers the reference policy's support, without importance sampling (Theorems IV.1–IV.2).
- GVPO extends to sequence-level on-policy distillation and a family of weighted reverse-KL objectives whose exact optimum remains the teacher, enabling cross-tokenizer distillation (Section V, Theorem V.1).
- Empirically GVPO outperforms eight post-training baselines (average 52.02 vs GRPO 46.69 on Qwen2.5-Math-7B) and six OPD baselines (35.99 vs PG 24.75 with a DeepSeek-R1-Distill-Llama-8B teacher).
- Both implicit regularization terms, Var(log πθ) and Cov(log πθ, log πθ′), are individually necessary for training stability (Table II).
Strengths
Elegant unification with a genuinely new partition-function cancellation trick
Section IV-A/B: the zero-sum weight condition (Eq. 6) lets the intractable Z(x) cancel in group comparisons, and the step-by-step derivation that the gradient equals that of an MSE between implicit- and actual-reward central distances gives the objective a transparent geometric meaning.
Unusually broad, like-for-like baseline coverage
Table I compares eight post-training algorithms (GRPO, Dr.GRPO, Remax, Reinforce++, GSPO, GMPO, GPG, Cispo) under identical budgets on Qwen2.5-Math-7B; Table VI adds six OPD baselines including GKD and MiniLLM in the same-vocabulary setting.
Multi-axis robustness checks beyond the headline table
Strong reproducibility signals
Honest catastrophic-failure ablation
Weaknesses
Theory is population-level; the trained estimator is never analyzed
Statistical rigor is uneven across experiments
Table II mixes evaluation scales
OPD gains partly entangled with a hand-tuned weighting
Incremental novelty as a journal extension
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This IEEE-style journal manuscript extends the authors' NeurIPS GVPO paper [31] into "GVPO++", adding an on-policy distillation (OPD) formulation, a family of weighted reverse-KL objectives, and expanded experiments. Central claims: (1) post-training methods share a gradient form — a weighted sum of response log-likelihood gradients (Section III, Eq. 4); (2) if a prompt group's weights sum to zero, the partition function Z(x) in the closed-form optimal policy (Eq. 2–3) cancels (Eq. 6), so the KL-constrained reward-maximization solution can be targeted without estimating Z(x) and without importance sampling; (3) the GVPO gradient equals that of an MSE between group-centered implicit rewards β·log(πθ/πθ′) and actual rewards (Section IV-B); (4) its unique optimum is exactly the KL-regularized optimum for any sampler covering the reference support (Theorems IV.1–IV.2), and n-step online use is equivalent to a β/n-regularized objective (Theorem IV.3); (5) the same machinery yields GVPD for sequence-level, cross-tokenizer OPD and extended weighted objectives (Theorem V.1). The claims are largely supported at the population-objective level, with the strongest empirical evidence on math reasoning.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GVPO and how is it different from GRPO?
GVPO (Group Variance Policy Optimization) is an LLM post-training algorithm that, like GRPO, samples a group of responses per prompt. Instead of GRPO's importance-sampling-weighted objective, it uses zero-sum weights so the partition function cancels, making its loss the mean squared error between implicit rewards (β·log of policy/reference ratio) and actual rewards within each group. The paper claims its unique optimum exactly solves the KL-constrained reward-maximization problem without importance sampling, whereas GRPO needs IS weights for off-policy reuse and carries no such optimality guarantee.
What results does GVPO++ report?
On Qwen2.5-Math-7B post-trained on Competition Math, GVPO averages 52.02 across five math benchmarks (AIME2024, AMC, MATH500, Minerva, OlympiadBench) versus 46.69 for GRPO and 49.31 for GSPO. As on-policy distillation into a Qwen2.5-Math-1.5B student, it averages 35.99 with a DeepSeek-R1-Distill-Llama-8B teacher versus 24.75 for policy gradient, and 35.35 versus MiniLLM's 26.22 with a Qwen-7B teacher. Note the headline comparisons are single runs; 10-seed variance is reported only at 1.5B against GRPO.
Can GVPO distill models with different tokenizers or vocabularies?
Yes. GVPO++ formulates on-policy distillation at the sequence level, needing only response-level log-probabilities of the student and teacher, so the two models need not share vocabulary or tokenizer. The paper demonstrates this by distilling from DeepSeek-R1-Distill-Llama-8B (Llama tokenizer) into Qwen2.5-Math-1.5B, where it clearly beats SFT, SeqKD and policy-gradient baselines.
Is GVPO's code available?
The paper states that the GVPO implementation has been integrated into the official verl GitHub repository, and Appendix A provides a minimal ~10-line implementation of the loss on top of verl's policy-loss interface, using the GRPO advantage computation without standard-deviation normalization.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3–7 are plots whose data points cannot be verified from the extracted text (axis labels partially garbled); quantitative claims tied to them (β/k/α/πs ablations, OPD length collapse) rest on figure appearance.
- Inline math is partially garbled by extraction (e.g., the Equation 10 decomposition and Appendix D reward formula); proofs were assessed as extracted.
- Table IV and Table VIII duplicate the same Reasoning Gym result set; treated as one experiment.
- Abstract contains an editorial typo ('exactly to the KL-constrained reward maximization objective'); treated as wording, not substance.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.