Beyond Quadratic Loss: The Stability Phase Diagram of Adam
Gaoxiang Tang, Huanran Chen, Ziming Liu
cs.LG · math.OC · stat.ML
Quick verdict
Adam loss spikes follow a near-linear (β1,β2) stability boundary; systematic evidence, but the key left-boundary law is an empirical single-seed 1D fit.
Summary
The paper charts when Adam(W) training exhibits macroscopic loss spikes as a function of its two momentum hyperparameters. Scanning logarithmically spaced (1−β1, 1−β2) grids across six small model-task settings (modular division/addition Transformers, Tiny Shakespeare, CIFAR-10 MLP and CNN, MNIST autoencoder), the authors detect macroscopic oscillations spectrally and find an approximately linear spiky/non-spiky boundary, 1−β2 = C(1−β1), with fitted exponents 0.75-1.06. A 1D quadratic loss instead yields a cubic boundary (γ ≃ 15.6δ^3.07), while 1D superquadratic losses |x|^n produce a wedge: an analytically derived right boundary CR = 2(n−2)/n (Theorem 3, recovering Bai et al. 2026a under a quiet-phase approximation) and an empirical left boundary CL ≃ 12(n−2)/(3n−4). The paper argues confident cross-entropy creates a 'core-wall' directional landscape: Hessian collapse gives a quadratic core (Lemma 5), exponential margin sums give a superquadratic wall, with core scale O(1/σa) (Theorem 7). Measured wall exponents (6.5-10.9, Table 1) inserted into CL(n) predict boundary coefficients ≈3.5-3.7. Limitations: the left-boundary law - the one actually applied to neural networks - is purely empirical and saturates as n grows; single seed; tiny scale; the Gaussian slope-distribution assumption is unvalidated; fitted prefactors (2.21-3.84) spread well beyond predicted CL.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An approximately linear boundary 1−β2 = C(1−β1) separates spiky from non-spiky Adam dynamics across six model-task settings, with fitted log-log exponents 0.75-1.06.
- One-dimensional quadratic loss yields a cubic boundary (γ ≃ 15.6δ^3.07), whereas superquadratic losses |x|^n yield a wedge with analytic right boundary CR = 2(n−2)/n (Theorem 3) and empirical left boundary CL ≃ 12(n−2)/(3n−4).
- Confident cross-entropy produces a core-wall directional landscape whose quadratic-core scale is O(1/σa) (Theorem 7), and pre-spike contraction of the core to the update scale constitutes finite-scale progressive sharpening.
- Measured wall exponents (nwall = 6.52-10.90, Table 1) fed into the empirical CL(n) relation predict the observed boundary coefficients (≈3.5-3.7).
Strengths
Systematic, well-documented phase-diagram methodology
Log-spaced (1−β1, 1−β2) grids across six model-task settings with all non-β hyperparameters frozen per setting, and a fully specified spectral period detector (Appendix B.2: log transform, third-order Butterworth forward-backward at 0.05 cycles/step, Welch spectrum, prominence ≥ 0.02·maxP).
Clean 1D isolation of the mechanism with controls
The quadratic (cubic boundary, γ ≃ 15.6δ^3.07) vs superquadratic (unit-slope wedge) contrast in Section 3 is sharp; Appendix C.1/Table 2 shows the cubic exponent is robust to k (3.064), η (3.210), and ϵ (3.009), and C.2 honestly reports that weak shifted weight decay preserves the left boundary but dissolves the wedge's lower edge.
Analytic right boundary consistent with concurrent work
Quantitative core-scale theory with one strong internal validation
Honest delimitation of what is derived vs fitted
Weaknesses
The left-boundary law - the one applied to neural networks - is a saturating curve fit
No replication or uncertainty quantification
Gaussian slope-distribution assumption unvalidated
Small-scale, full-batch settings only
Detection hyperparameters fixed without sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Beyond Quadratic Loss - The Stability Phase Diagram of Adam
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What causes loss spikes in Adam training according to this paper?
The paper argues spikes occur when the second-moment memory (set by β2) decays too slowly relative to the first-moment memory (β1) - an approximately linear boundary 1−β2 = C(1−β1) separates spiky from stable regimes - combined with a superquadratic 'core-wall' loss landscape that confident cross-entropy creates at the scale of a single Adam update.
How should I choose β1 and β2 to avoid loss spikes?
The measured phase diagrams suggest keeping 1−β2 above roughly C·(1−β1) with C ≈ 2-4 for the studied small models; for example, with β1 = 0.9 this puts the boundary near β2 ≈ 0.98-0.996. These constants come from small-scale, single-seed experiments, so treat them as guidance rather than a guarantee for large-scale training.
What is the core-wall loss landscape?
Along the Adam-preconditioned gradient direction, the loss near a minimum first grows quadratically (a flat 'core' caused by Hessian collapse in confident cross-entropy), then transitions into a much steeper-than-quadratic 'wall' caused by exponential margin sums, followed by an outer rollover. The core width is predicted to scale as 1/σa, where σa is the spread of margin slopes along the direction.
Is the Adam stability boundary in this paper derived theoretically or fitted empirically?
Mixed. The right boundary of the spiky wedge in 1D superquadratic models is derived analytically (CR = 2(n−2)/n, matching Bai et al. 2026a). The left boundary - the one that matches the neural-network experiments - is an empirical power-law fit, CL ≃ 12(n−2)/(3n−4), with no derivation; the paper states this openly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures could not be visually inspected from the extracted text; phase-diagram structure, boundary-fit quality, and the Figure 9 slice progression are taken from captions and in-text numbers only.
- The paper reports no goodness-of-fit statistics (R², residuals, intervals) for the power-law fits, so fit quality cannot be assessed even in principle from the text.
- The core-wall evidence in the main text is worked out in detail for one model, one beta pair, and one step (modular division, step 37,210); the other five settings appear only as Table 1 means.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.