Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
cs.AI · cs.LG
Quick verdict
Empirical scaling law for grokking time in MLPs (D dominates H) is valuable, but internal inconsistencies and censored-data fitting weaken confidence.
Summary
The paper maps when two-hidden-layer MLPs grok on modular arithmetic (addition mod 113, division mod 97) via a full-factorial sweep of 384 configurations, varying width H, data fraction D, learning rate η, and AdamW weight decay λ. Fitting log-linear models on the 297 grokked runs out of 356 completed (28 diverged), it reports T_grok ∝ H^-0.27 D^-2.04 η^-0.50 λ^-0.64 (R²=0.732; 0.821 with four significant pairwise interactions), concluding data fraction dominates width about eight-fold in exponent (≈4× vs ≈1.2× speedup per doubling). It further claims a sharp phase boundary near λ≳1.0 above which nearly all configurations grok, and monotonic weight-norm compression during transitions (median ratio 0.42 over 14 runs), formalized as two conjectures. Robustness checks are unusually thorough: threshold sensitivity (D exponent −2.15 to −1.88 for 85–99% thresholds), per-task fits, a Weibull AFT model over all 356 runs (concordance 0.71), and leave-one-level-out CV (R² 0.67–0.76). Key limitations: the main fit uses one seed per configuration (rerun CV ~8%, 68% near the boundary); 59 right-censored non-grokking runs are excluded from OLS although censoring correlates with λ; text and Figure 7c give contradictory norm correlations (ρ=0.06 vs 0.63); and 'structurally suppressed' claims rest on a 150K-step budget. Code is stated available.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
4 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Grokking onset time follows T_grok ∝ H^-0.27 D^-2.04 η^-0.50 λ^-0.64 across 297 grokked MLP runs, with data fraction the dominant lever (~4x per doubling) and width the weakest (~1.2x).
- A sharp phase boundary near λ ≳ 1.0 in (η, λ)-space separates grokking from non-grokking configurations.
- Weight norms compress monotonically through the transition (median ratio 0.42 over 14 runs), supporting norm-based accounts, with a conjectured configuration-independent threshold r* in 0.3–0.5.
- Data coverage and weight decay act as partially substitutable drivers of the transition (log D × log λ interaction +0.38, t=5.3), both destabilizing the memorizing solution.
Strengths
Full-factorial sweep at meaningful scale
384 configurations across two tasks (add mod 113, division mod 97), three widths, four levels each of D/η/λ; 356 completed runs, 297 grokked — the scale is what makes exponent estimation for Tgrok possible at all (Secs. 2–3).
Honest, multi-pronged robustness reporting
Exponent SEs, Spearman ρ, threshold sensitivity (D exponent −2.15→−1.88 across 85–99% thresholds, Table 3), leave-one-run-out R²=0.799, leave-one-level-out CV (R² 0.67–0.76, median multiplicative error 1.4–1.6×, Table 4), and a Weibull AFT over all 356 runs including 59 censored (Sec. 3.3, concordance 0.71).
Quantifies folklore into a falsifiable form
Pre-committed falsifiable predictions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks *when* overparameterized networks cross from memorization to generalization (grokking), restricted to two-hidden-layer MLPs with learned embeddings on addition mod 113 and division mod 97 (Sec. 2). From a full-factorial sweep of 384 configurations (2 tasks × 3 widths × 4 data fractions × 4 learning rates × 4 weight decays; 356 completed, 297 grokked), it claims: (1) a power law for grokking onset, T_grok ∝ H^-0.27 D^-2.04 η^-0.50 λ^-0.64 (R²=0.732; 0.821 with interactions, Sec. 3); (2) a sharp phase boundary at λ ≳ 1.0 (Sec. 4, Conjecture 1); (3) weight-norm compression (median ratio 0.42, n=14) as a regime indicator (Sec. 5, Conjecture 2). The headline — data coverage, not capacity, dominates transition timing — is positioned against Omnigrok (Liu et al. 2023) and Nanda et al. (2023), converting qualitative knowns into fitted exponents.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 0 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is grokking in neural networks?
Grokking is the phenomenon where a network first perfectly memorizes its training data, then — after much more training, typically with weight decay — abruptly switches to generalizing to unseen inputs. This paper studies it on modular arithmetic tasks where the generalizing solution is known to be a Fourier-style circuit.
Does model size speed up grokking according to this paper?
Only weakly. Doubling width cuts grokking time by ~1.2x (exponent -0.27), while doubling the training-data fraction cuts it ~4x (exponent -2.04). Weight decay (-0.64) and learning rate (-0.50) sit in between. The paper argues data coverage, not capacity, is the main lever.
Is the code and data available?
The paper states code and run logs for all 384 configurations are available at github.com/anishesg/icml-papers. Note the main fits use a single random seed per configuration, and the authors report seed variability of ~8% (up to 68% near the phase boundary), so exact numbers may vary on re-runs.
Do the results apply to transformers or LLMs?
Not established. The experiments are two-hidden-layer MLPs on modular arithmetic. The authors explicitly list testing transformers as future work and caution that the exponents may differ for other architectures and tasks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels are known only through extracted text labels and captions; visual content (e.g., which series Fig. 4 actually plots) could not be inspected directly. The body text appears complete through Appendix D (p. 9), so no truncation is assumed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.