Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang, Luke Zettlemoyer
cs.LG · cs.CL
Quick verdict
Claims MoEs overfit faster than dense under data repetition, driven by total params; convincing small-scale grid study, though single-seed, not frontier scale.
Summary
The paper empirically studies how data repetition interacts with Mixture-of-Experts (MoE) sparsity in language-model pretraining. Across three compute-matched active-parameter scales (80M, 200M, 1B; up to 8.5B total parameters), the authors train dense and MoE transformers on OLMoE-mix domains (DCLM web, StarCoder code, peS2o academic, Wikipedia), fixing total tokens at roughly 20 per active parameter while varying unique tokens so repetition rates R range from 1 to 1024 and beyond. MoEs degrade earlier and faster than dense models: degradation begins at R=4 versus R=8 for dense, and the MoE advantage on unique data reverses at R=32. Overfitting severity tracks total rather than active parameters and holds across single domains, data filtering, and token-to-parameter ratios, though mixing repeated domains with unique, semantically similar data (DCLM + peS2o) dampens the effect. Among regularizers, only dropout and FFN/expert output masking mitigate overfitting—strong masking lets MoEs beat dense even at R>64—while weight decay, gradient clipping, and router jitter do not. Mechanistically, routers ossify early, expert specialization (knockout cost) rises with R, and dropout reduces specialization without changing routing stability. Main limitations: most runs are single-seed with variance checked only at 80M, downstream accuracies are near chance, the mechanistic account is correlational, and no code release is stated, so conclusions may not transfer to frontier scale.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MoE LMs degrade earlier and more severely under data repetition than compute-matched dense LMs: degradation begins at R=4 versus R=8, and the MoE unique-data advantage reverses at R=32 (§3.1, Fig 1).
- The severity of repetition-induced overfitting is governed by total rather than active parameter count (§3.1, Fig 4).
- Dropout and output-masking regularization substantially mitigate repetition overfitting — with strong masking MoEs beat dense even at R>64 — but weight decay, gradient clipping, and router jitter have no effect, and no method matches all-unique data (§4, Fig 8, 16).
- MoE routers ossify early in training and expert specialization (knockout cost) grows with repetition; dropout reduces specialization without altering routing stability, implying overfitting is not primarily router-driven (§5, Fig 9–10).
Strengths
Cleanly controlled repetition design
Unique-token subsets are nested via one fixed data permutation (App. A.5), removing set-composition confounds; total tokens are compute-matched at T≈20·Na and MoE active parameters are matched via g=1/k (§2.1, §3).
Unusual breadth of the grid
Three scales (80M/200M/1B active), MoE expert counts up to 256 and granularities 1/32–1/2 (Fig 4, Table 1), R up to 1024 and extended to ≈2^15–2^20 at 80M (App. B.2), four single domains (Fig 3), filtering interpolation (Fig 6), and two-domain mixes with per-domain repetition (Fig 7).
Honest null results and robustness checks
Precisely defined mechanistic metrics with an informative dissociation
Weaknesses
Small scale and short runs vs. frontier framing
Thin statistical reporting
Total-parameter claim is suggestive, not isolated
Downstream evaluation too weak; loss/accuracy divergence undiscussed
Citation and presentation hygiene errors
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how data repetition interacts with model sparsity in language-model pretraining. Central claims: (1) MoEs degrade earlier and faster than compute-matched dense models under repetition (§3.1, Figure 1: degradation appears at R=4 for MoE vs R=8 for dense; the unique-data advantage reverses at R=32); (2) repetition-induced overfitting is governed by total rather than active parameters (Figure 4; the 200M dense model, 193.9M total, sits between 80M-active MoE(32×1/4) at 157.7M and MoE(64×1/4) at 244.4M total); (3) dropout and masking-based regularizers mitigate but never fully cure (§4, Figure 8: with strong masking, MoEs beat dense even at R>64); (4) routers ossify early and expert specialization correlates with overfitting (§5, Figures 9–10). The claims are well supported inside the studied regime (80M–1B active parameters, T≈20·Na) but are framed for frontier training, where the evidence is only indirect.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do Mixture-of-Experts models overfit more when training data is repeated?
Yes, according to this paper: compute-matched MoE language models begin degrading at 4× data repetition versus 8× for dense models, and lose their unique-data advantage after 32 repetitions. The effect grows with total (not active) parameter count and appears across web, code, academic, and encyclopedic data.
Can regularization fix MoE overfitting from repeated data?
Partially. Dropout and masking-based methods (FFN/expert output masking) substantially mitigate it — with strong masking, MoEs beat dense models even at 64× repetition — but no method matches training on all-unique data, and weight decay, gradient clipping, and router jitter showed no effect.
Are the conclusions valid for large production LLMs?
The study covers 80M to 1B active parameters (up to 8.5B total) trained on at most 20B tokens. The authors argue patterns depend on total parameters, but direct evidence at frontier scale is not provided, and the paper itself cites work showing repetition damage grows with scale.
Why do MoEs overfit more to repeated data?
The paper's mechanistic analysis suggests routers fix their token-to-expert assignments early in training, so each expert updates on a small, near-stationary subset of tokens. Expert specialization, measured by the loss increase when an expert is knocked out, rises with repetition, and dropout reduces it without changing routing stability.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure plots are extracted as interleaved axis/label text, so exact in-figure curve values and some legends could not be read; numeric statements above rely on the body text, captions, and Table 3.
- Bold 'default' markings in the Appendix A.2 hyperparameter table are not preserved by extraction, so the exact default weight-decay value for §3 runs cannot be confirmed from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.