UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Robert Hu, Carlo Luschi, Paul Balanca
cs.LG
Quick verdict
Proposes UE5M3 FP4 block scaling for RHT-free pretraining; single-seed 8B runs edge NVFP4 on loss, well-validated emulator, but head-to-head is confounded.
Summary
This Graphcore technical report replaces NVFP4's signed E4M3 block scales with unsigned E5M3 scales, whose wider range permits FP4 pretraining with periodic (every-50-step) sample-and-hold tensor scaling, no randomized Hadamard transform, and FP4 in all 112 eligible linears. The team pretrained Nemotron-H 8B for 188.7B tokens on 32 GB200 ranks using a software 'probe-matched' FP4 GEMM emulator fitted to native Blackwell outputs (64-product groups, round-toward-zero cross-group adds, 1/1024 lattice snap), validated via 258 permutation witnesses and exact gradient parity on a 1.29B-parameter model. The proposed recipe reaches final-window training loss 2.3090 vs 2.3474 for Transformer Engine NVFP4 (BF16: 2.2651); held-out NLL 2.32230 vs 2.32592 (lower at 8/12 checkpoints); downstream gains of +0.13/+1.01/+0.01 points. Notably, UE5M3 with full TE settings achieves the best FP4 loss (2.2902), suggesting the scale format matters more than the simplifications. A native ablation removing RHT and the BF16 exemption raises model-body throughput 21.2% but destabilizes training. Key limitations: a single seed-42 trajectory per configuration, substituted OLMo data and a 5.3x shorter horizon than NVIDIA's 1T-token setup, margins at or below seed-noise level, a software-vs-native execution-path confound, and no hardware support for UE5M3.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Pairing E2M1 payloads with unsigned E5M3 block scales permits stable FP4 pretraining using periodic D=50 sample-and-hold tensor scaling, without RHT and without a BF16 final-block exemption.
- The proposed block-16 recipe beats Transformer Engine NVFP4 at 188.7B tokens: final-window training loss 2.3090 vs 2.3474, step-30,000 held-out NLL 2.32230 vs 2.32592, and higher point estimates on all three downstream aggregates.
- UE5M3 with full TE settings achieves the best FP4 training loss (2.2902), implying the wider scale format is the dominant stabilizer rather than the recipe simplifications.
- A native NVFP4 ablation jointly removing RHT and the BF16 exemption raises measured model-body throughput 21.2% (3,212 to 3,894 tokens/s), though that configuration destabilizes training.
- A software probe-matched emulator exactly reproduces tested native NVFP4 GEMM outputs, including full 1.29B-model forward/backward gradient parity and 100-step AdamW raw-storage parity.
Strengths
Exemplary emulator validation
The probe-matched GEMM model (64-product FP32 partials, RTZ cross-group adds, 1/1024 ties-to-even snap) is identified via 258 permutation witnesses plus 7 BF16-boundary witnesses (App. B, Tables 15-17), passes 512-cell broad probes, yields exact logits/loss and all 1,291,929,600 gradients on a 1.29B model, and 100-step AdamW byte parity with a single 2-ULP gradient element (App. D, Table 21). Most software-FP4 studies assert emulation fidelity; this one measures it.
Honest reporting discipline
Sec. 7.1 explicitly labels single-trajectory comparisons as descriptive; Table 8 itemizes exact/partial/substituted fields vs NVIDIA's setup; App. A shows the inference-scale policy choice changes NLL by under 9.0e-5; the UE5M3-with-TE-settings control outperforming the proposed recipe (2.2902 vs 2.3090) is reported prominently rather than buried.
Useful range theory with a matched control
Both quality and cost measured
Weaknesses
No seed replication; margins at noise level
Confounded headline comparison
Partial baseline fidelity
Proposed recipe never runs on hardware
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Review: UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is UE5M3 FP4?
A 4-bit microscaling format for neural network training: 4-bit E2M1 payload values are grouped in blocks of 16, each sharing one 8-bit unsigned E5M3 scale. Because scales are never negative, UE5M3 reuses E4M3's unused sign bit as a fifth exponent bit, giving about 137x more representable scale range (largest code 61,440 vs 448) at the same local spacing.
Does UE5M3 FP4 beat NVIDIA's NVFP4 for pretraining?
In this paper's experiments, yes, but by small margins on a single training run per method: final training loss 2.3090 vs 2.3474 (or 2.2902 for UE5M3 with NVIDIA's full settings), held-out NLL 2.32230 vs 2.32592, and downstream gains of +0.13/+1.01/+0.01 percentage points on OLMES Core 9, MMLU, and MMLU-Pro. The authors themselves note these are single-seed descriptive comparisons, so independent replication is still needed.
Is code available for this paper?
Yes. The reproducibility statement (Appendix E) points to github.com/MrHuff/ue5m3-fp4 for the portable UE5M3 FP4 reference implementation. Note that reproducing the 8B pretraining results requires 32 GB200 GPUs and the authors' internal OLMo-family data mixture; the small-scale emulator probes and regression controls are CPU-feasible.
What does the 21.2% throughput improvement mean?
It does not refer to the proposed UE5M3 method (which has no hardware support yet). It is a measurement of native NVFP4 execution on one GB200 when RHT and the BF16 exemption for 16 final-block projections are removed: model-body forward/backward throughput rose from 3,212 to 3,894 tokens/s. That faster configuration actually trained worse, with repeated loss spikes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are embedded in the extracted text as interleaved axis labels and legends, so visual trajectory details beyond the quoted numbers and tables could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.