Hyperparameter Scaling Laws Across MoE Sparsity
Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou
cs.LG · cs.AI · cs.CL
Quick verdict
1,800-run study deriving unified MoE hyperparameter scaling laws with activation-ratio correction; credible and rigorous but one backbone, one holdout.
Summary
This paper from Ant Group's Ling Team empirically derives unified hyperparameter scaling laws for ultra-sparse Mixture-of-Experts (MoE) language models. Based on 1,800 pre-training runs spanning six activated-parameter scales (10M–324M), up to 6B total non-embedding parameters, roughly 20 trillion tokens, and about 200,000 H800 GPU-hours, the authors show that neither activated nor total parameter count alone explains shifts in the optimal learning rate (LR) and batch size (BS) across sparsity levels. They propose h*(X,A) = kh·X^γh·A^δh, where optimal LR follows compute C=MD and optimal BS follows training tokens D, with activation ratio A as a multiplicative correction (δη=0.136, δB=−0.084). Grouped leave-one-ratio-out and leave-one-scale-out cross-validation show the multiplicative form beats scale-only and additive alternatives, though it is statistically indistinguishable from a log-interaction family. On a single held-out 12B-total-parameter, A=1/64 target (D=159B tokens), the frozen law predicts hyperparameters closer to observed optima than refitted DeepSeek and Step laws (loss gaps 1.03–1.37‰ vs 0). Controls confirm transfer across expert granularities. Main limitations: a single backbone and optimizer (hybrid linear-attention/MLA, Muon), no seed replication, one extrapolation target, and no code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Neither activated parameter count nor total parameter count alone explains the shifts in optimal learning rate and batch size across MoE sparsity levels; the activation ratio A must be modeled explicitly (Figure 2).
- At fixed sparsity, the optimal learning rate follows a power law in compute C=MD (insensitive to M/D allocation) and the optimal batch size follows a power law in training tokens D (Figures 3–4).
- Across sparsity levels, A enters both laws as a multiplicative prefactor with δη=0.136>0 and δB=−0.084<0 (Table 2), yielding h*(X,A)=kh·X^γh·A^δh.
- On a frozen 12B-total-parameter, A=1/64 holdout (D=159B, C=3×10²⁰ FLOPs), the law jointly extrapolates beyond the fitting ranges and predicts hyperparameters closer to observed optima than published or refitted DeepSeek and Step laws (Table 4).
- The activation-ratio relation transfers across capacity-matched expert granularities and cannot be reduced to absolute expert count or total capacity (Section 4.4, Figure 9).
Strengths
Unmatched experimental scale with genuinely controlled design
1,800 runs over six activated scales (10M–324M) × four activation ratios, ~20T tokens, ~200k H800 GPU-hours (Table 7). Figure 2 matches activated N and separately total Ntot, cleanly demonstrating the sparsity shift survives both alignments — the paper's strongest causal evidence.
Disciplined, leakage-aware validation protocol
LOAO/LONO grouped cross-validation built at the loss-surface level (Sec. 4.2), prespecified 0.1% near-optimal threshold following Bi et al. (2024), and held-out losses frozen before predictions were committed (Sec. 4.3, Table 4).
Resolves conflicting prior findings with controlled variable comparison
Mechanistic backing for the empirical signs
Candid limitations section
Weaknesses
Single training stack limits portability
Functional-form selection is not statistically decisive
Thin practical margin on the headline extrapolation
One extrapolation target, N held at the boundary
No code or data release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (Ant Group Ling Team, arXiv 2609.08690) asks how the optimal peak learning rate (η*) and global token batch size (B*) vary with MoE sparsity, formalized through the activation ratio A = Eact/Etot (Sec. 2.1). Four central claims: (1) neither activated N nor total Ntot parameter count explains η*/B* shifts across sparsity at matched scale (Fig. 2); (2) at fixed A, η* follows a power law in compute C=MD and B* in training tokens D (Figs. 3–4); (3) A enters as a multiplicative prefactor h*(X,A)=kh·X^γh·A^δh with δη=0.136>0 and δB=−0.084<0 (Table 2); (4) the law jointly extrapolates to a frozen 12B-total-parameter, A=1/64 holdout where it beats published and refitted DeepSeek and Step laws (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That optimal learning rate and batch size for MoE training depend on the expert activation ratio A, not just on model size, compute, or token count. The paper fits unified laws h*(X,A)=kh·X^γh·A^δh: the optimal learning rate rises with compute and with A (δ=0.136), while the optimal batch size rises with training tokens and falls as the model gets sparser (δ=−0.084).
How large was the experimental study?
1,800 pre-training runs across six activated-parameter scales (10M–324M) and four sparsity levels, up to 6B total non-embedding parameters, processing about 20 trillion tokens at roughly 200,000 equivalent H800 GPU-hours, plus one held-out 12B-total-parameter MoE with only 1/64 of experts activated.
Can I apply these scaling laws to my own model training?
Treat them as a template rather than universal constants. The laws were fit on one specific stack (hybrid linear-attention/MLA backbone, Muon optimizer, WSD schedule, sigmoid routing), and the numerical coefficients depend on units and the training setup. The qualitative directions (sparser MoE → lower LR, larger batch) transferred across tested expert granularities, but you would need small-scale probes to refit coefficients for your own configuration.
Is code or data available for this paper?
No. The paper does not mention any code or data release, and the pre-training data mixture is internal to Ant Group, so independent verification requires comparable GPU-scale resources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.