Information-Time Proximal Policy Optimization
Yongcheng Zeng, Xinyu Cui, Yan Song, Guoqing Liu, Hongsheng Xin, Kaike Zhang, Cheng Deng, Kun Zhan, et al.
cs.LG · cs.AI
Quick verdict
InfoPPO re-times PPO by information density; coherent theory and consistent Qwen3 math gains, but single-seed evals and theory-practice gaps temper the claims.
Summary
InfoPPO targets the tension in LLM reinforcement learning between long-range credit assignment and effective-horizon control caused by token-level discounting. The authors reparameterize the MDP's temporal axis using state-wise information density, instantiated as the normalized next-token predictive entropy of the frozen old policy, so discounting and GAE trace decay accumulate over information rather than token count. The same density regulates policy updates through adaptive clipping with bounds 1+log(1+ε_high·ρ) and its reciprocal, replacing fixed PPO clipping. Theoretically, the paper extends the performance difference lemma to information time, derives a policy-improvement lower bound (Theorem 4.2) for updates whose per-state total-variation movement scales with ρ, and bounds the perturbation from recomputing the clock across iterations (Propositions 4.3–4.5), with proofs in the appendix. Training Qwen3-4B/8B/14B on DAPO-Math-17K, InfoPPO averages 34.9/41.2/49.4 Mean@16 across five math benchmarks (AMC23, AIME24/25/26, BeyondAIME), beating PPO, DAPO, and DAPO-FT, with response lengths comparable to DAPO rather than inflated. Information-time PPO stays stable under γ=0.95–0.999 and λ=0.95–0.99, settings where token-time PPO collapses (e.g., length saturation at γ=0.999); a Llama-3.2-3B replication shows smaller gains. Main limitations: single-run experiments without significance testing, a heuristic (undirected) log-scaled clipping form, a gap between the one-step softmax theory and multi-epoch clipped training, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reparameterizing MDP time by normalized predictive entropy of the old policy restores useful non-trivial discounting (γ<1) and trace decay (λ<1) in long-horizon LLM RL, where token-time PPO collapses.
- Policy updates whose per-state total-variation movement scales with information density satisfy a policy-improvement lower bound (4Cα²/(1−γ)²) for the information-time MDP.
- Adaptive clipping that widens with information density improves sample efficiency while preserving proximal control, yielding consistent gains over PPO, DAPO, and DAPO-FT across Qwen3-4B/8B/14B without inflating response length.
- Cross-iteration recomputation of the information clock perturbs return comparisons by at most 2Rmax·ε(r)/e, so clock drift does not invalidate the analysis (Proposition 4.5).
Strengths
One quantity unifies credit propagation and update control
The same ρ(st) drives information-time GAE (Eq. 23), adaptive clipping (Eq. 25), and the divergence constraint in the theory (Eq. 14), giving the method an unusually coherent structure rather than a bag of tricks.
Complete theoretical treatment with proofs
Lemma 4.1 extends the performance difference lemma to information time; Theorem 4.2 gives a 4Cα²/(1−γ)² improvement bound under the state-wise TV constraint; Propositions 4.3–4.5 handle the softmax-step divergence scaling (β̄CHmax/2·ρ) and the 8Rmaxα/e clock-recomputation perturbation, all with derivations in Appendix A.
Sharp mechanism-isolating experiments
Consistent gains without length inflation
Design choices are ablated, not just asserted
Weaknesses
No statistical treatment of results
Theory-practice gap in the update analysis
Confounded main comparison
Entropy as information density is a semantic leap
Baseline breadth and strength
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
InfoPPO — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is InfoPPO in one sentence?
InfoPPO is a variant of PPO for LLM reinforcement learning that measures the MDP's time axis by accumulated information — the normalized next-token entropy of the frozen old policy — instead of raw token count, and uses that same density both for discounting/GAE credit propagation and for per-token adaptive clipping of policy updates.
Why does discounting hurt PPO when training LLMs on math reasoning?
With outcome-only rewards, per-token discounting multiplies the terminal reward by γ raised to the sequence length (thousands of tokens), so the learning signal decays to nearly zero; turning discounting off restores the signal but loses control of the effective horizon, and the paper shows token-time PPO with γ=0.999 drives response lengths to saturate near the maximum generation length.
What results does the InfoPPO paper report?
Training Qwen3-4B/8B/14B Base on DAPO-Math-17K, InfoPPO reaches average Mean@16 scores of 34.9, 41.2, and 49.4 across AMC23, AIME24, AIME25, AIME26, and BeyondAIME, versus best baselines (PPO, DAPO, DAPO-FT) of 32.9, 36.3, and 46.6, with response lengths similar to DAPO rather than longer; it also stays stable for γ from 0.95 to 0.999 and λ from 0.95 to 0.99 where token-time PPO degrades sharply.
Is InfoPPO code publicly available?
The paper text reviewed here contains no code or data release statement; hyperparameters and training details are documented in Appendix B, but independent reproduction would require building on an external RLVR training framework.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–9 could not be analyzed visually from the extracted text; figure-dependent claims (training curves, length saturation at γ=0.999, ablation orderings) rest on captions and body prose only.
- Appendix D consists largely of verbatim model reasoning traces accompanying Figures 10–15, which pad the text but carry little independently checkable signal.
- The abstract supplied in the task metadata is truncated mid-sentence; the full-text abstract was used instead.
- No code or data availability statement appears in the extracted text, so codeAvailable=false reflects absence of a claim, not a confirmed negative.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.