Structured Features Overfit Where Random Features Grok
Chon-Fai Kam, Miloud Bessafi, Frederic Cadet
cs.LG · stat.ML
Quick verdict
Structured Fourier features remove the grokking delay XVS prove for random ones; masking shows active-mode count drives collapse; solid but one λ, one target.
Summary
The paper tests the generality of the grokking phenomenon that Xu, Vardi and Safran (XVS) proved for over-parameterized ridge regression over random Gaussian features, where the memorize-then-generalize delay grows as 1/λ in the weight decay. On a band-limited Fourier (group-character) feature map over Z_97^2 with a linear readout, the authors fit the representable single-mode target a−b at fixed weight decay λ=10^-2 on n=3763 of the 9409 modular pairs. Enlarging the band radius M degrades peak held-out accuracy monotonically from 0.999 (M=18, q/n=0.364) to 0.066 (M=40, near the 1/97 chance level), with degradation already present at q/n=0.638, below the interpolation threshold; no memorize-then-generalize regime appears anywhere in the sweep. A masking experiment at fixed nominal dimension M=32 shows that restricting the active support to 1089 modes restores perfect generalization (1.000±0.000) while the full 4225-mode band collapses to 0.185±0.006, so active-mode count rather than capacity ratio controls the outcome, acting through the teacher-weighted spectrum of the empirical Gram matrix (Eq. 3). Appendix A carefully refutes the superficially grokking-like M=24 transient. Main limitations: a single weight decay, a single sample size, one single-mode target, three seeds with peak-envelope metrics, a heuristic-only collapse threshold, and no matched Gaussian-feature control run.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On a band-limited Fourier feature map over Z_97^2, the representable single-mode target a−b at fixed weight decay λ=10^-2 shows peak held-out accuracy falling monotonically from 0.999 (M=18) to 0.066 (M=40) as the band grows, with no memorize-then-generalize regime anywhere in the sweep.
- The degradation begins at capacity ratio q/n = 0.638, below the interpolation threshold, so it is not an interpolation effect and is distinct from double descent.
- At fixed nominal dimension (M=32), masking the band to 1089 active modes restores held-out accuracy of 1.000 with zero seed variance while the full 4225-mode band collapses to 0.185, so active-mode count rather than capacity ratio governs the outcome.
- The mechanism is the teacher-weighted spectral measure of the empirical Gram matrix (Eq. 3): the collapse is a property of the ridge fixed point, hence not a finite-training-budget artifact.
- Together these results delimit the reach of the XVS grokking theorem by exhibiting a structured feature map, at fixed realizability and over-parameterization, on which the memorize-then-generalize delay does not appear.
Strengths
Training-free realizability audit
Because the features are group characters, the spectral gap δ² (Eq. 2, Appendix B) decides expressibility in closed form with no training run: Table 1 shows δ² ~ 10⁻¹²–10⁻¹⁷ for representable targets versus δ² = 1 for the full-plane target ab. This enables holding the XVS theorem's hypothesis (realizability) fixed while varying only feature geometry — the exact comparison the research question demands and one impossible with Gaussian features.
Cleanly controlled masking experiment
At fixed nominal dimension M=32 (4225 modes instantiated), restricting active modes to k=16 (1089 modes) yields 1.000 ± 0.000 held-out accuracy while the full band collapses to 0.185 ± 0.006 (Table 3). The two independent routes to 4225 active modes (masked and unmasked) agree exactly, and masked-1089 sits beside unmasked-1369 within 0.001 (Appendix C table) — a strong internal consistency check.
Rigorous disposal of the apparent-grokking point
Mechanism with a falsifiable consequence
Unusually transparent protocol and self-scoping
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Features Overfit Where Random Features Grok — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is grokking and what does this paper claim about it?
Grokking is when a model fits its training data long before it starts generalizing. A 2026 theorem by Xu, Vardi and Safran proves such a memorize-then-generalize delay for over-parameterized ridge regression with random Gaussian features, with the delay growing as 1/λ in the weight decay. This paper shows the delay disappears on a structured feature map: with band-limited Fourier features on modular arithmetic over Z_97^2, widening the frequency band for a provably learnable target makes held-out accuracy fall monotonically from 0.999 to 0.066, with no delayed transition anywhere at fixed weight decay λ=10^-2.
Is the collapse reported in this paper just double descent?
The authors argue no. Degradation starts at capacity ratio q/n = 0.638 (band radius M=24), below the interpolation threshold q/n=1, and a masking experiment at fixed nominal dimension restores perfect accuracy (1.000 with 1089 active modes vs 0.185 with all 4225 modes active). The outcome therefore tracks the number of actively used Fourier modes - acting through the teacher-weighted spectrum of the empirical Gram matrix - rather than the parameter-to-data ratio that drives double descent.
Can the experiments in this paper be reproduced?
In principle yes. Everything is synthetic and CPU-scale: p=97, n=3763 training pairs, full-batch gradient descent with learning rate 0.05, initialization scale ν=5, seeds 0-2, and 30k-60k step budgets are all specified in Appendix D. However, code is only 'available upon request' rather than released, so reproduction currently means reimplementing from the text.
What are the main caveats of this study?
All dynamical results use one weight decay (λ=10^-2), one sample size (n=3763), one single-mode target (a−b) and three seeds; the collapse threshold is a heuristic bracket (between 2401 and 3249 active modes) rather than a derived formula; and the contrast with random features, which do grok per prior work, is taken from the literature rather than run under matched experimental conditions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as captions; plotted curves could not be inspected visually, so judgments rest on the tabulated values in Tables 1-3.
- All quantitative results are self-reported; no independent verification of the Appendix D numbers was possible from this text.
- The zero-ridge (λ→0) observations and the promised c(λ) derivation are deferred to a 'full-length version' that is not part of this document.
- The comparison baseline (Xu, Vardi and Safran, ICML 2026) is cited rather than reproduced; its exact experimental protocol could not be checked here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.