Towards Full Pipeline FP8 Reinforcement Learning for LLMs
Fanchao Chen, Ziheng Jiang, Ziyun Wei, Zheng Zhong, Du Li, Chi Zhang, Haibin Lin, Shivaram Venkataraman
cs.LG · cs.AI
Quick verdict
FP8 ratio noise causes over-clipping and entropy collapse; Calibrated Clipping restores near-BF16 scores—strong analysis, but evidence is single-run.
Summary
The paper explains why full-pipeline FP8 reinforcement learning (FP8 rollout plus FP8 training) destabilizes LLM training even with truncated importance sampling (TIS). Training Qwen3-8B-Base with GRPO on DeepScaleR, the authors observe entropy surges after roughly 100 steps and garbled outputs under tensorwise, rowwise, and blockwise FP8 scaling. Using BF16 shadow passes, they show that small per-probability quantization errors (NMAE under 3%) are amplified 1.7x-2.9x in the importance ratio, over-clipping negative-advantage tokens at the lower bound (~70% of late-training over-clipping; ~90% from high-entropy garbled responses; clip fraction ~4x BF16), so gradients that should penalize pathological outputs vanish. Their fix, Calibrated Clipping, periodically (every 20 steps) aligns the FP8 lower clipping bound to the BF16 reference clipping quantile and rebalances the upper bound to preserve the BF16 positive/negative update ratio. Across GRPO (Qwen3-8B, Qwen2.5-32B) and DAPO (Qwen3-14B), calibrated FP8 largely restores baseline scores — e.g., 8B rowwise average 47.0 to 56.5 vs BF16 57.6, blockwise 58.6 — though calibrated tensorwise/rowwise stay 1-2 points below. Relaxing the lower bound alone prevents surges but over-penalizes; CISPO and BAPO collapse in FP8 settings. Training throughput reaches up to 1.5x BF16 (tensorwise), though the benchmark excludes the method's BF16 reference passes. Limitations: single runs, best-checkpoint selection, no variance reporting, dense models only, no code release stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Full-pipeline FP8 RL suffers mid-training entropy surges and performance collapse even with TIS correction, across tensorwise, rowwise, and blockwise scaling.
- The instability is caused by compounded FP8 quantization noise distorting the importance ratio, which disproportionately over-clips negative-advantage tokens at the lower bound and zeroes the gradients meant to penalize pathological outputs.
- Calibrated Clipping — quantile-matched lower bound plus update-ratio-rebalanced upper bound against a BF16 reference, recalibrated every 20 steps — eliminates the surges and restores near-BF16 scores across GRPO and DAPO at 8B-32B scale.
- Alternative remedies fail in the FP8 regime: naive lower-bound relaxation over-penalizes (Figure 5), and CISPO/BAPO collapse (Appendices A.2/A.3).
- Full-pipeline FP8 training with the method achieves up to 1.5x BF16 training throughput (tensorwise; blockwise 10-20%).
Strengths
Novel, precisely-measured failure mode
Section 3 quantifies ratio-noise amplification (1.7x-2.9x NMAE vs single probabilities, Figure 3), asymmetric lower-bound over-clipping (~70% of late-training over-clipping at 1-eps, ~90% sourced from high-entropy responses, Figure 4), and ~4x clip fraction vs BF16 even under blockwise scaling (Figure 1) — a mechanism no prior FP8-RL work (FlashRL, Jet-RL, Unified FP8, FP8-RL) identified.
Method grounded in explicit reference statistics, not heuristics
Algorithm 1 derives the lower bound by CDF quantile matching over negative-advantage tokens against BF16 and solves the upper bound to preserve the BF16 positive/negative update ratio rho; smoothing caps (deltaL=0.05, deltaH=0.1) and 20-step amortization make it practical, and sensitivity to interval/init is reported (A.5).
Verification goes beyond correlation
Broad empirical matrix
Head-to-head with the nearest competitors
Weaknesses
No variance or statistical testing
Efficiency claim excludes the method's own overhead
Recovery is incomplete and the baseline reversal unexplained
Causal isolation incomplete
Scope and availability limits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies full-pipeline FP8 reinforcement learning for LLMs — FP8 rollout and FP8 training — and claims instability persists even with truncated importance sampling (TIS). The central claim is mechanistic: compounded FP8 quantization noise distorts the token-level importance ratio, disproportionately pushing negative-advantage tokens below the clipping lower bound (1−ε), zeroing the gradients that should suppress pathological outputs, which then accumulate into mid-training entropy surges. The remedy, Calibrated Clipping, re-derives the PPO clipping bounds from a high-precision BF16 reference. Claims are scoped to token-level clipping in GRPO/DAPO with dense Qwen models (8B–32B) on math/coding RLVR; the authors explicitly defer sequence-level (GSPO) and MoE settings to future work (Section 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does this paper solve?
Full-pipeline FP8 reinforcement learning for LLMs — where both rollout (generation) and training run in FP8 — suffers sudden mid-training entropy surges and garbled outputs that destroy performance. The paper shows the cause: FP8 quantization noise is amplified in the PPO importance ratio, wrongly clipping negative-advantage tokens at the lower bound so the model never learns to suppress pathological outputs.
What is Calibrated Clipping and how does it work?
It is a dynamic PPO clipping method for FP8 RL. Every 20 training steps, two forward-only BF16 reference passes are run on the current batch; the FP8 clipping lower bound is set so the fraction of clipped negative-advantage tokens matches the BF16 reference (quantile matching), and the upper bound is adjusted so the positive-to-negative update ratio also matches BF16. Bound updates are smoothed and capped to avoid abrupt trust-region shifts.
How much faster is FP8 RL training than BF16?
The paper reports up to 1.5x the BF16 training throughput with tensorwise FP8 scaling (10-20% with blockwise) across 8B-32B models at 4K-16K context, on top of roughly 30% generation speedup from FP8 rollout. Note this benchmark excludes the periodic BF16 reference passes that the proposed method requires, so end-to-end gains are somewhat lower.
Does calibrated FP8 fully match BF16 performance?
Mostly, but not exactly. On Qwen3-8B, calibrated blockwise FP8 slightly exceeds BF16 (average 58.6 vs 57.6 across eight math benchmarks), while calibrated tensorwise and rowwise remain about 1-2 points below (55.9 and 56.5). On Qwen2.5-32B and in DAPO experiments on AIME24, calibrated variants recover most of the loss and come within a few points of the BF16 baseline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only via OCR'd axis labels and captions; exact curve shapes and values (entropy trajectories, throughput bars in Figure 7, clipping-bound trajectories) cannot be independently verified from the extracted text.
- Key multipliers (1.7x-2.9x NMAE amplification, ~70%/~90% over-clipping proportions) are read from figure annotations; underlying per-token data is not available.
- Main text and appendices appear complete with no obvious truncation, so the review covers all reported experiments, but figure-level claims inherit OCR uncertainty.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.