Small Enough to Know Everything: The Fully-Enumerable Transformer as an Instrument for the Science of Delayed Generalization
cs.LG
Quick verdict
Tiny-transformer 'grokking instrument' claim put to a preregistered 4000x scale test: 2 of 3 laws hold to 50M; rigorous, honest, limited to enumerable tasks.
Summary
The paper argues that tiny transformers trained on fully-enumerable tasks (about 100 input cells) constitute a scientific instrument for studying grokking, with four capabilities: exact generalization ceilings, invariant-preserving task surgery, exhaustive weight observation, and cheap multi-seed survival statistics. Its empirical core is a preregistered 'conservation study': three laws previously established at 12K parameters — an exact recoverability ceiling, a role-conflict delay law, and a weight-decay response — are re-measured from scratch at 12K, 1M, and 50M parameters (360 runs plus a 44-run control arm, n=12 seeds per cell, verdict rules frozen in advance). The ceiling law is conserved (0/144 Holm-corrected binomial violations; grok rates order monotonically in the ceiling at every scale), and the role-conflict law is conserved (Spearman rho >= 0.75 at every scale, permutation p < 1e-4), with an informative scale map: larger models tolerate role conflict less, a deficit surviving learning-rate sweeps and a tripled budget. The weight-decay law is deformed: its descending arm is absent even in the from-scratch 12K arm, demoting the prior inverted-U to a protocol artifact. Limitations: tasks stay enumerable at all scales (only the model scales), a single architecture family and optimizer, n=12 power limits, a three-point weight-decay grid, and a small (n=3) CPU control.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Tiny transformers on fully-enumerable tasks form a scientific instrument with four capabilities — exact falsifiable generalization ceilings, invariant-preserving task surgery, full-weight observation, and survival statistics over many seeds.
- Two of three laws established at 12K parameters (the exact recoverability ceiling and the role-conflict delay ordering) are conserved across a 4,000x scale span to 50M parameters under an identical, preregistered from-scratch protocol.
- Larger models are less, not more, tolerant of role conflict per training epoch, and this 50M deficit survives learning-rate adjustment and a tripled budget.
- The weight-decay response law deforms systematically: its descending arm is protocol-dependent, absent even in the from-scratch 12K replication, demoting the inverted-U to a property of the prior training setting.
- Task-side quantities computed by enumeration retain their predictive force once the model outgrows the instrument's regime — the license needed to treat the enumerable setting as a model organism.
Strengths
Preregistered verdict rules with demonstrated teeth
Design document with frozen three-way verdict criteria was committed (with hash) before any cross-scale run (Section 3.3); the study then reports a failed prediction (L1 attainment decline, Section 4.1) and demotes its own prior weight-decay 'law' (Section 4.3), showing the criteria were not cosmetic.
Statistical hygiene above field norms
Exact binomial tests Holm-corrected across all 144 runs (0 ceiling violations, Section 4.1); permutation tests (10^5 permutations) explicitly replacing the asymptotic normal approximation under censoring-induced ties (Section 4.2); Wilson intervals throughout; a sustained grokking criterion that kills multiple-looks inflation on the 30-cell test set.
Rate-level science instead of seed-level anecdotes
Confound controls behind the headline L2 scale map
Honest self-demotion of prior results
Weaknesses
The harder generalization question is deferred by design
Everything rests on self-cited prior work
Structural power limits
Narrow protocol coverage
Preregistration is self-attested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This single-author preprint (arXiv 2609.20166v2, cs.LG) makes two claims. Methodologically, it argues that tiny transformers on fully-enumerable tasks (~100-cell input tables) should be treated as a scientific instrument for delayed generalization, with four capabilities: (a) computed, falsifiable generalization ceilings; (b) invariant-preserving task surgery; (c) exhaustive weight observation; (d) survival-time statistics over many seeds (Section 2). Empirically, it runs a preregistered 'conservation study' (Section 3): three laws from the author's own 12K-parameter prior work — L1 exact recoverability ceiling, L2 role-conflict delay, L3 weight-decay response — re-measured from scratch at 12K, 1M, and 50M parameters (4,000x span) under identical protocol: same task tables, per-seed splits, tokenization, AdamW (lr 3e-3, wd 0.1), batch 16, 15K-epoch budget, n=12 seeds per cell, with three-way verdict rules frozen in a hashed design document before any cross-scale run.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is grokking?
Grokking is the phenomenon where a neural network suddenly generalizes to unseen data long after it has already memorized the training set. This paper studies it in tiny transformers trained on small arithmetic-style tasks where every possible input can be exhaustively evaluated.
What does 'conservation' mean in this paper?
A law measured at very small model scale (12K parameters) is called conserved if it still holds — under the identical training protocol and preregistered criteria — at much larger scales (1M and 50M parameters, a 4,000x span). Two of three tested laws were conserved; the weight-decay law deformed.
Did the weight-decay 'inverted-U' law survive the study?
No. Its descending arm did not appear at any scale, including a from-scratch replication at the original 12K scale, so the authors conclude the inverted U was a property of their earlier training setup rather than a law of the task family.
Do these results apply to large language models?
Not directly. The tasks remain small enough to fully enumerate at every scale; only the model grows. The paper's claim is limited to task-side computed quantities continuing to predict as the model scales, and it explicitly leaves non-enumerable tasks to future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are only available as caption text here; visual claims (monotone orderings, KM curves, ceiling-tracking scatter) were assessed from the reported numbers, not the graphics.
- This is an arXiv preprint (v2), not a peer-reviewed publication.
- Several load-bearing references (the author's prior laws [2], the grokking-clock theory [7], the measurement-validity audit [6]) are 2026 preprints whose content could not be verified from this text.
- The per-seed JSON data release is stated but not verifiable here; explicit analysis-code release is not stated in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.