Bellman Policy Optimization
Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing
cs.LG · cs.CL · math.OC
Quick verdict
BPO derives a critic-free trajectory-level reformulation of Policy Mirror Descent, proves equivalence to PMD on reachable states, and beats four GRPO-family losses on AIME, but the shipped loss departs from the proved objective and only one un-replicated run is reported.
Summary
Bellman Policy Optimization (BPO) is a critic-free policy-optimization method for reinforcement learning with verifiable rewards (RLVR), derived from Policy Mirror Descent (PMD). Directly applying PMD to language generation needs advantage estimates at every intermediate state, which normally requires a value model. BPO avoids this by exploiting the terminal-reward structure: because intermediate rewards are zero, the Bellman equations give A_mu(s_t, y_t) = V_mu(s_{t+1}) - V_mu(s_t), so advantages telescope along a response to sum to R(x,y) - V_mu(x) (Eq. 23). Substituting this into the PMD optimality condition yields a trajectory-level squared-residual objective (Eq. 18) that depends only on the verifier reward and the initial value V_mu(x), the latter estimated as the mean reward of sampled responses. Theorem 1 proves that this objective and PMD (Eq. 16) share the same unique optimal solution on states reachable under the rollout policy; the necessity direction is proved in Appendix A.1 via a martingale argument over token-level residuals. The practical loss (Eq. 14) keeps the GRPO form but replaces the token-level importance-sampling ratio with a mismatch-correction weight omega = (1 + eps - mu(y_t|s_t)) / (1 + eps - pi(y_t|s_t)), capped at C and combined with the GRPO clipping mask. It is reached through four approximations: linearising the squared residual around pi = mu, estimating V_mu and the prompt weight phi = 1/std from each group of G responses, replacing the full reverse KL with binary KL (giving the identity in Eq. 27), and additive smoothing. Experiments train Qwen3-30B-A3B-Base on the English subset of DAPO-Math-17k for 400 steps (256 prompts x 16 responses, 16384-token limit). BPO reaches a peak mean Avg@32 of 50.5% on AIME24-26 versus 39.5% (GRPO-ClipHigher), 43.5% (GSPO), 47.4% (CISPO) and 46.4% (DPPO). Main limitations: a single run with no variance estimates, best-checkpoint reporting, one base model and one dataset, and no analysis of the gap between the proved objective and the implemented loss.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 6 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- For autoregressive generation with terminal rewards, the Bellman equations make token advantages telescope, so a trajectory-level PMD objective can be written using only the terminal reward and the initial value, avoiding any value model.
- The critic-free objective (18) and the original advantage-based PMD objective (16) have the same unique optimal solution on states reachable under the rollout policy (Theorem 1), for any positive prompt weighting phi.
- Approximating that objective yields a practical per-token loss that keeps the GRPO form but replaces the importance-sampling ratio with a smoothed mismatch-correction weight omega = (1+eps-mu)/(1+eps-pi).
- On Qwen3-30B-A3B-Base trained on DAPO-Math-17k for 400 steps, BPO reaches a peak mean Avg@32 of 50.5% on AIME 2024-2026, exceeding GRPO-ClipHigher, GSPO, CISPO and DPPO by 11.0, 7.0, 3.1 and 4.1 percentage points.
- BPO is insensitive to its two hyperparameters: on Qwen3-4B-Base it stays between 24.1% and 25.8% across eps in {0.05, 0.1, 0.2, 0.3} and between 25.3% and 25.8% across C in {2.0, 3.0, 4.0}, all above the 20.5% GRPO-ClipHigher baseline.
Strengths
Clean and genuinely useful telescoping identity
Section 3.1 Step 2 shows that with zero intermediate rewards, A_mu(s_t,y_t) = V_mu(s_{t+1}) - V_mu(s_t), so the per-trajectory sum collapses to R(x,y) - V_mu(x) (Eq. 23). This reduces the whole credit-assignment problem to the terminal reward plus a single scalar per prompt, and V_mu(x) is estimable from the same rollouts already being collected.
Complete equivalence theorem with a non-trivial proof
Theorem 1 is proved in both directions. Sufficiency follows from Eq. 24; necessity (Appendix A.1) constructs D_t = 1{t<=tau} d_pi(S_t,Y_t), shows E[D_t | F_{t-1}] = 0 using Eq. 10, so M_n is a martingale, and from M_Tmax = 0 a.s. plus bounded horizon deduces every token-level residual vanishes. The step from 'terminal residual zero' to 'every partial sum zero' is exactly the subtle part, and it is handled correctly.
Honest, well-controlled experimental protocol
Loss reduces to a one-line change over GRPO
Gains hold at end of training, not only at the selected peak
Weaknesses
The proved objective is not the objective that is optimised
Single run, no variance, best-checkpoint reporting
Narrow empirical scope
Component attribution is not isolated
Ablations are on a different, smaller model and against one baseline
No release of code and no cost accounting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Bellman Policy Optimization (BPO) targets a specific, well-circumscribed problem: how to run Policy Mirror Descent on an LLM without a critic. PMD (Section 2.5) updates the policy with an action-value term plus a divergence penalty, and instantiating it with f = A_mu gives the per-state problem (16) whose closed-form solution is pi+(y|s) = mu(y|s) exp(eta A_mu(s,y)) / Z_mu(s). Applying (16) to language generation needs A_mu at every intermediate state, which in practice means training a value model — expensive in memory and, as the authors note citing [22], unreliable on reasoning tasks.
The paper's three stated contributions are: a critic-free trajectory-level reformulation of PMD; a proof that it has the same unique optimal solution as PMD on states reachable under the rollout policy; and a practical loss derived from it, evaluated on mathematical reasoning. The scope is deliberately narrow — one algorithm, one equivalence theorem, one benchmark family — and the claims are stated with appropriate precision (equivalence holds on states reachable under mu, for any positive prompt weighting phi).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 6 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Bellman Policy Optimization and how does it differ from GRPO?
BPO is a policy-optimization loss for reinforcement learning with verifiable rewards on LLMs. It is derived from Policy Mirror Descent rather than from PPO-style clipping. The key trick is that with only a terminal reward, the Bellman equations make token-level advantages telescope along a response, so their sum equals the final reward minus the prompt's expected reward. That removes the need for a value model. In implementation BPO looks almost identical to GRPO: same group-normalised advantage, same clipping mask, but the token-level importance-sampling ratio pi/mu is replaced by a mismatch-correction weight (1+eps-mu)/(1+eps-pi), capped and smoothed.
What accuracy does BPO achieve on AIME?
Training Qwen3-30B-A3B-Base on the English subset of DAPO-Math-17k for 400 steps, BPO reaches a peak average accuracy of 50.5% Avg@32 across AIME 2024, 2025 and 2026 (57.4 on AIME24, 41.0 on AIME25, 53.0 on AIME26). The baselines under identical settings score 39.5% for GRPO-ClipHigher, 43.5% for GSPO, 47.4% for CISPO and 46.4% for DPPO. At the end of training BPO is at 49.4% versus 45.5% for the strongest baseline.
Does BPO need a value model or critic?
No. That is the central design goal. The trajectory-level objective only needs the verifier's terminal reward and the initial value V_mu(x), which is the expected reward for a prompt under the rollout policy and is estimated as the mean reward of the sampled responses for that prompt. Values or advantages at intermediate states are never estimated, which removes the memory and compute cost of a separate critic and avoids inaccurate learned value estimates on reasoning tasks.
Is BPO sensitive to its two hyperparameters?
Ablations on Qwen3-4B-Base with 1000 training steps show little sensitivity. Varying the smoothing parameter eps over 0.05, 0.1, 0.2 and 0.3 gives average accuracies of 25.8%, 25.4%, 25.5% and 24.1%, and varying the truncation cap C over 2.0, 3.0 and 4.0 gives 25.3%, 25.4% and 25.8%; the GRPO-ClipHigher baseline in the same setup scores 20.5%. The main 30B experiments use eps = 0.1 and C = 3.0. Note that these ablations are on a smaller model than the headline experiment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are described only through their flattened captions and the accompanying tables; all numeric claims here come from Tables 1-3 and the text, not from reading the curves.
- Appendices A.1 and A.2 (the proofs of Theorem 1 and of identity 27) are present in full, but the main-text proof sketch for necessity is compressed; the review's reading of the martingale argument follows the appendix.
- Baseline configurations (GSPO clipping interval, DPPO threshold, CISPO cap) are inherited from other papers as stated in Appendix B and were not independently tuned by the authors.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.