Optimizer Memory Schedules for Outscaling the Overtraining Axis
cs.LG
Quick verdict
Empirically rich OT-axis study (51M-253M, 1-256x) showing rankings shift with horizon; ADANA outscales AdamW per DANA theory; claims partly fit-dependent.
Summary
The paper asks whether optimizer rankings and optimal hyperparameters change as language models are trained far beyond the compute-optimal horizon. The authors train decoder-only Transformers at 51M, 124M, and 253M parameters with AdamW, ADANA, Muon, and SOAP across overtraining factors reaching 256x on the smallest model, sweeping the base learning rate independently at every setting. They find the preferred learning-rate schedule reverses along the overtraining axis (linear better at 1x, cosine at 8x and above), the best weight-decay coefficient scales approximately as sqrt(OT), and longer horizons favor longer fixed momentum memory. Tuning memory and learning rate jointly per horizon improves fixed-memory optimizers but does not remove ADANA's scaling advantage. With log-time weight decay and a newly proposed momentum-cooldown rule, ADANA outscales AdamW with an equivalent-OT exponent of 1.15-1.20, close to the 2-kappa=1.15 prediction from DANA theory on power-law random features. Muon and SOAP instead maintain roughly constant token multipliers of about 1.4-1.7x and 1.3-1.7x over AdamW, with SOAP possibly improving further at the highest factors. ADANA starts behind both but closes the gap, surpassing Muon and becoming competitive with SOAP at the highest tested factors. Main limitations: a single seed per configuration, memory tuning only on the 51M model, and high-OT comparisons that rely on extrapolated scaling fits.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Optimizer rankings, the preferred LR schedule, and optimal weight decay and memory change substantially across the overtraining axis, with the best WD coefficient scaling approximately as sqrt(OT) (fitted exponents 0.527 uniform, 0.473 log-time).
- With log-time weight decay and momentum cooldown, ADANA outscales AdamW with equivalent-OT exponents of 1.15-1.20 across three model sizes, close to the DANA power-law-random-features prediction 2-kappa=1.15.
- Muon and SOAP provide approximately constant token multipliers of 1.4-1.7x and 1.3-1.7x over AdamW across most of the measured range, with SOAP possibly gaining further at the highest OT factors (~1.9x at 51M/128x).
- ADANA starts behind both matrix-preconditioned optimizers (0.59-0.69x token multiplier at 2x OT) but closes the gap with training, surpassing Muon and becoming competitive with SOAP at the highest tested OT factors.
- Per-horizon tuning of AdamW's fixed memory improves it (3.3 mpt at 128x OT) but does not eliminate ADANA's scaling advantage.
Strengths
Unmatched coverage of the overtraining axis
Four optimizers x three model sizes (51M/124M/253M) with OT factors up to 256x/32x/8x, producing 76 uniform-WD loss values (Appendix G.1), each cell backed by an independent base-LR sweep — far beyond prior single-horizon comparisons (Wen et al. 2025 capped at 8x-16x).
Explicit sweep-hygiene criteria
Appendix B.5 defines a boundary-interiority rule (minimum and all points within one millipoint must lie off tested boundaries), applied to nested sweeps; incomplete sweeps are flagged as open circles in Figures 17-18, 23, 27.
Fit-level model selection and uncertainty honesty
Controlled fair comparisons
Reproduction-grade implementation detail
Weaknesses
Single seed per configuration
Headline high-OT claims depend on extrapolated fits
Controls limited to the smallest model
ADANA hyperparameters transferred, not retuned per horizon
Unresolved batch-size confound for ADANA
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether optimizer rankings and optimal hyperparameters survive when models train along the overtraining (OT) axis — far beyond the Chinchilla-optimal ~20 tokens/parameter. Four optimizers are compared on decoder-only Llama-style Transformers (SwiGLU/RMSNorm/QK-RMSNorm) at 51M, 124M, and 253M nominal parameters, with OT factors up to 256x (51M), 32x (124M), and 8x (253M) on FineWeb, batch 256x2,048 tokens; the base learning rate is swept independently at every (optimizer, model, OT) cell, yielding 76 uniform-WD losses for the primary fit (Appendix G.1). Claims: (1) treatments must scale with horizon — the preferred LR schedule reverses (linear best at 1x, cosine at 8x/32x, Appendix D), the best weight-decay coefficient scales approximately as sqrt(OT) (fitted exponents 0.527 uniform / 0.473 log-time; adopted rules c=8*sqrt(f) and c=2*sqrt(f), Appendix E.4), and longer horizons favor longer fixed memory (optimal M rises from 20 to 640-1,280 across 1x-128x, Figures 3/16); (2) with log-time weight decay plus a new momentum-cooldown rule, ADANA outscales AdamW at equivalent-OT exponents 1.15-1.20, near the DANA power-law-random-features prediction 2-kappa=1.15 (Figure 24); (3) Muon and SOAP give roughly constant 1.4-1.7x and 1.3-1.7x token multipliers over AdamW (strongest-treatment comparison, Figure 8), SOAP reaching ~1.9x at 51M/128x; (4) ADANA starts behind both (0.59-0.69x at 2x OT, Section 7.3) and closes the gap at the highest OT factors.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is overtraining and why does it matter for optimizer choice?
Overtraining (OT) means training a model on many more tokens than the compute-optimal ratio of roughly 20 tokens per parameter (Chinchilla); inference-cost economics often favors it. This paper shows that which optimizer wins, and which hyperparameters are best, change as the OT factor grows from 1x up to 256x — so comparisons done at a single training horizon can be misleading.
Does ADANA beat Muon and SOAP?
Not uniformly. At short horizons ADANA trails both (token multipliers of 0.59-0.63x vs Muon and 0.63-0.69x vs SOAP at 2x OT). As training extends it catches up, surpassing Muon and becoming competitive with SOAP at the highest tested overtraining factors (up to 256x on the 51M model), while the authors caution that rankings beyond their measured range remain uncertain.
How should weight decay be set when training much longer?
The paper finds the best weight-decay coefficient scales approximately as the square root of the overtraining factor for both uniform and log-time schedules — adopted rules c_uniform(f)=8*sqrt(f) and c_log(f)=2*sqrt(f) on their setup, with fitted exponents of 0.527 and 0.473, extending the Power-Lines result of Bergsma et al. to more optimizers and to log-time decay.
What is momentum cooldown in this paper?
A rule the authors propose for scheduled-memory optimizers: during terminal learning-rate decay, the momentum timescale is capped at the local LR-decay timescale (tau = max(1, min(momentum timescale, LR decay timescale))), preventing stale long-memory averages from dominating the final phase. It substantially improves ADANA at long horizons, has little effect on fixed-memory AdamW, and compounds with log-time weight decay.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked [...sampling gap...] regions (e.g., parts of Sections 5-6, Appendix A, and portions of E/G/H); all judgments above rely on visible text and none rests solely on gap content.
- No code- or data-release statement for this paper appears in the sampled text; codeAvailable=false reflects the excerpt, and a release note may exist in unsampled sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.