Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui, et al.
cs.LG · cs.AI
Quick verdict
Identifies 'Value Flattening' in PPO critics and shows SP3O's 3-anchor sparse supervision helps, with strong diagnostics and ablations but single-seed, terminal-reward-only evidence.
Summary
The paper identifies 'Value Flattening': in PPO for LLM reasoning, critic predictions change little across states within a response while Monte Carlo (MC) estimates of state values, averaged over many sampled continuations, shift sharply. The phenomenon is shown on Qwen3-4B-Base trained on DAPO-Math-17k and reproduced in stochastic FrozenLake, where flattening intensifies as maze size grows. The authors attribute it to (1) an implicit variance penalty: with terminal-only rewards and gamma=lambda=1, the per-token MSE critic loss decomposes into a response-mean fit plus a term penalizing within-response prediction variance; and (2) redundant updates, since adjacent token states are nearly identical and yield highly aligned critic gradients. They propose SP3O, which applies the critic loss only at three well-separated anchors (0.3/0.6/0.9 relative positions, plus 0.95 for responses of at least 6144 tokens). SP3O reduces profile MSE versus MC values by 36%/11%/21% at 30/60/90% response progress, and improves the Qwen3-4B math average from 37.60 (PPO) and 39.26 (GRPO) to 45.57, with OOD average rising from 51.95 to 59.28; 8B gains are smaller. Ablations support K=3 and spaced placement. Limitations: single-seed results without significance tests, binary terminal rewards only (KL=0, gamma=1), Qwen3 models only, and diagnostic panels based on 64 responses.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PPO critic predictions remain comparatively flat across states within a response while Monte Carlo estimates of policy-conditioned state values change sharply, a phenomenon the authors name Value Flattening.
- Value Flattening also appears in stochastic FrozenLake and becomes more pronounced as the state space grows.
- The effect is driven by an implicit within-response variance penalty in the dense critic MSE loss (under shared terminal targets) and by redundant, highly aligned updates from temporally correlated neighboring states.
- Applying the critic loss at only three well-separated anchors (SP3O) mitigates Value Flattening and improves policy performance, e.g., raising the Qwen3-4B seven-task math average from 37.60 (PPO) to 45.57.
Strengths
Well-characterized new failure mode with dual evidence
Value Flattening is demonstrated both in LLM training (Figure 1: critic profiles flat across checkpoints 50-350 for both correct and incorrect responses) and in a controlled stochastic FrozenLake experiment (Figure 2b,c) where prediction smoothness worsens monotonically as maze size grows from 32 to 128 under a fixed training configuration.
Exact mechanistic decomposition, not just correlation
Section 4.2 and Appendix A.2 derive the identity (v_mean-R)^2 + within-response variance (Eq. 5, Eq. 16-17), showing the dense critic loss literally contains a variance penalty, complemented by gradient/representation measurements (Figure 3: high gradient cosine that decays with token-position separation).
Simple intervention with consistent gains and thorough ablations
Theory anticipates the standard objection
Reusable diagnostic methodology
Weaknesses
No training-run variance or significance testing
Narrow setting behind broad claims
Diagnostic ground truth is itself an estimate on a small panel
Method bundles two components; gains may partly come from generation stability
Modest 8B margins over GRPO
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Value Flattening in PPO critics?
Value Flattening is a failure mode where a PPO critic's value predictions barely change across states within a single LLM response, even though the true state values (estimated by averaging terminal rewards over many continuations sampled from the same state) change sharply between adjacent reasoning steps. The paper demonstrates it in math-reasoning RL training and in a FrozenLake control environment, where it worsens as the state space grows.
What is SP3O and how does it work?
SP3O (SParse Proximal Policy Optimization) modifies PPO by applying the critic's value loss at only three well-separated positions in each response (30%, 60%, and 90% of the way through, plus an extra 95% anchor for long responses) instead of at every token. This avoids an implicit penalty that pushes predictions toward a flat value profile and reduces redundant updates from nearly identical neighboring token states. The actor objective and rollouts are unchanged.
How much does SP3O improve over PPO and GRPO?
On Qwen3-4B-Base trained on DAPO-Math-17k, SP3O raises the average score over seven math benchmarks from 37.60 (PPO) and 39.26 (GRPO) to 45.57, and the six-task out-of-distribution average from 51.95 (PPO) and 56.44 (GRPO) to 59.28. On Qwen3-8B-Base the gains are smaller: 50.51 vs 48.50 (PPO) in-domain. Results are single-run averages over 32 (math) or 4 (OOD) generations.
Why does PPO's critic loss flatten value predictions within a response?
With terminal-only rewards, every token position in a response is regressed toward the same observed return. The mean-squared-error loss then decomposes exactly into a term fitting the response-mean prediction to that outcome plus the empirical variance of predictions within the response. The second term directly penalizes any within-response variation, so minimizing the loss actively pushes the critic toward flat predictions, independent of what the true value profile looks like.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through extracted captions and axis-label text; figure-level claims (e.g., exact curve shapes, variance bands) rely on captions rather than inspectable plots.
- Appendix A.2 derivations were extracted but are notation-heavy; intermediate algebraic steps could not be fully verified from the text.
- Paper metadata lists 8 authors while the embedded full text lists 12; the discrepancy is noted but does not affect the technical review.
- The linked code repository (github.com/Dodojordi/SP3O) was not inspected, so repro-readiness is based on stated hyperparameters and released-code claim only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.