An Exploratory Replica-Overlap Probe of the Grokking Transition
cs.LG · cond-mat.dis-nn · cs.AI
Quick verdict
Pre-registered RSB-overlap probe of grokking self-reports UNDETERMINED: defective alignment invalidates its instrument; an honest failed-instrument report.
Summary
The paper reports a pre-registered, 64-replica probe of whether the distribution of pairwise weight overlaps — a replica-symmetry-breaking (RSB)-inspired order parameter — changes across the grokking transition in modular-addition networks (p=113). Sixteen independently seeded networks were trained in each of four configurations (train fractions 0.25/0.30/0.40 plus a random-label control) to sustained convergence or a 40,000-epoch ceiling. The registered analysis never reaches a verdict: a post-hoc audit found the Git Re-Basin-style alignment permutes weights but not the corresponding biases or attention head-internal symmetries, so it does not preserve the network function and every weight-space overlap q_wt is invalid. The registered fp32/fp64 precision rail also failed for 100% of the 120 frac40 pairs (mean discrepancy 2.8×10⁻⁵ against a 10⁻⁶ tolerance), yielding registered outcome UNDETERMINED (C0_INSTRUMENT_INVALID). Only frac40 cleared the 12/16 checkpoint gate. A post-hoc Hartigan-dip interval on the same data contains zero (95% CI [−0.017, 0.034]) but the test has zero power at simulated separations, so the interval is uninformative; the overlap SD rises about 5.6× (0.00444→0.02471), the only statistic with power at the observed effect. Seven protocol deviations (D1–D7) are logged, including unexecuted controls and a missing pre-analysis power calibration. Grokking rates of 0/16, 11/16, 16/16 are descriptive only, confounded with split identity. This is a transparent failed-instrument report; main limitations are the invalid instrument, a single task and architecture, and no public code statement.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The registered alignment implementation permutes weights without the corresponding biases and head-internal permutations, does not preserve the network function, and therefore invalidates every weight-space overlap value in the study.
- The pre-registered decision rule returns UNDETERMINED (C0_INSTRUMENT_INVALID): these data provide neither a confirmatory null nor a validated reading of the Parisi order parameter for grokking.
- For frac40, the post-hoc Hartigan-dip interval contains zero (95% CI [−0.017, 0.034]) but the test has zero power at the simulated separations, making the interval uninformative; the overlap SD increase of ≈5.6× is the only statistic with power at the observed effect.
- Grokking rates of 0/16, 11/16 and 16/16 across the three fraction configurations remain descriptive because train fraction is completely confounded with split identity.
- A valid successor test requires function-preserving alignment (including ReLU rescaling and within-head Q/K, V/O freedoms), matched-epoch controls, and a pre-specified power calibration.
Strengths
Registered outcome reported as UNDETERMINED rather than spun
Despite a null-looking dip interval and a striking SD effect, the paper refuses confirmatory language (Abstract, §8), assigns reason code C0_INSTRUMENT_INVALID, and withholds frac25/frac30 as RAISE instead of folding them into a null on RSB (§5.1) — the gate architecture worked exactly as registered.
Locatable self-audit of the instrument defect
§3/D5 pinpoints apply_permutation() permuting only nine *.weight tensors, quantifies the unpermuted biases (median max|b_i| = 0.382, 0.084, 0.020 for mlp_in.bias, W_O.bias, mlp_out.bias), and explains why the synthetic positive control (overlap restored to 1.000000) could never detect it.
Post-hoc power calibration correctly reframes the null
Timestamped deviation ledger D1–D7
Intellectual self-correction
Weaknesses
Every analysed measurement inherits an invalid instrument
Fraction and split identity fully confounded
Registered controls never executed (D3)
Precision rail mis-designed inside the registration
No public code or data statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This pre-registered study asks whether the distribution of pairwise weight overlaps P(q_wt) between independently trained replicas — a replica-symmetry-breaking (RSB)-inspired order parameter — changes across the grokking transition on modular addition (p=113, Power et al. 2022 task). Sixty-four networks ran in four 16-replica configurations (frac25, frac30, frac40, plus a matched-split random-label control nogrok) to sustained convergence or a 40,000-epoch ceiling. The central claim is deliberately deflationary: the registered decision rule returns UNDETERMINED (C0_INSTRUMENT_INVALID), and the authors explicitly disclaim both a confirmatory null and any validated reading of the Parisi order parameter (Abstract, §8). Secondary exploratory claims: a post-hoc Hartigan-dip interval containing zero that is uninformative because the test has zero power (D4); an overlap standard-deviation ratio of about 5.6; and a descriptive 0/16, 11/16, 16/16 grokking-rate pattern that is confounded with split identity (§5.1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did this paper find evidence of replica symmetry breaking during grokking?
No. Its registered outcome is UNDETERMINED (reason code C0_INSTRUMENT_INVALID): the weight-alignment code did not preserve the network function, so the overlap data cannot support or refute an RSB reading of grokking in either direction.
What went wrong in the experiment?
The permutation-alignment step permuted weight matrices but not bias vectors or attention head-internal symmetries, so the aligned networks were not functionally equivalent to the originals. In addition, the fp32/fp64 precision check failed for all 120 analysed pairs, and several registered controls were never run (deviations D1–D7).
Is the reported 5.6× increase in overlap spread a real discovery?
It is the only statistic the paper's own calibration shows has statistical power at the observed effect, but it was computed through the defective alignment, so the authors themselves treat it as uninterpretable rather than as an RSB signature.
What would a valid version of this study require?
Function-preserving alignment that also handles biases, attention head-internal permutations and ReLU rescaling symmetries; matched-epoch random-label controls to rule out drift from continued training and weight decay; and a power calibration run before the analysis, not after.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are available only as captions in the extracted text; the KDEs, accuracy distributions and bootstrap intervals could not be inspected visually.
- The pre-registration document itself is not included; all deviation claims rest on the authors' own audit, quoted commit hashes and timestamps.
- Training hyperparameters and the full architecture are deferred to unpublished pre-registration §B2, so reproduction from the paper text alone is not possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.