Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance
Sai Niranjan Ramachandran, Suvrit Sra
cs.LG · cond-mat.dis-nn · cond-mat.stat-mech · cs.AI
Quick verdict
Models SGD collapse as symmetry-forced percolation with discrete scale invariance; ambitious, hedged theory, but empirical cascades rest on 2-4 point fits.
Summary
The paper models SGD training as a percolation process. Building on stochastic-collapse results (Chen et al. 2023), it treats architectural permutation symmetries as affine invariant sets that trap trajectories, defines an attractor-equivalence relation over subnetwork path measures, projects the classes onto a Reeb graph, and renormalizes it (assuming a fast/slow timescale separation) into a monotone percolation graph whose order parameter is the fractional size of the largest component. Because Sn symmetry binds n indistinguishable subnetworks simultaneously, merges occur in blocks of size n rather than by single-edge attachment; the authors derive a jump ΔO=(n−1)i/N, characterize when it survives the thermodynamic limit (only for Θ(N) components, Corollary C.18), and show the resulting microtransition densities form a geometric cascade with magnification factor λ(n)=n^σ — a discrete scale invariance. Relative variance of the order parameter is proposed as a precursor diagnostic. The trapping mechanism is extended to Adam/AdamW under a heavy-tailed noise model with explicit stopping times and preliminary diagnostics. Empirically: kinematic and K=6 SGD toy models match λ≈2.00 (a two-point fit); transformer grokking shows a 3-peak cascade (λ=2.11, FPR=0.1%); UCI and vision results are mixed (Heart Disease λ=1.71; German Credit fails as negative control, FPR=80.2%), and several settings yield too few peaks for any cascade. Main limitations: the theory assumes exact affine symmetry and block decoupling while measurements use a soft spectral effective-rank proxy bridged by a heuristic with two free parameters; key fits rest on 2-4 points; no code statement appears in the provided text.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Architectural permutation symmetry forces SGD's collapse toward invariant sets to proceed by simultaneous n-body block merges rather than single-edge attachment, formalized as a percolation process over a renormalized Reeb graph.
- Order-parameter jumps ΔO=(n−1)i/N survive the thermodynamic limit only when merging components are macroscopic (Θ(N)); for microscopic merges the transition is asymptotically continuous.
- The discrete block-merge constraint yields Discrete Scale Invariance: successive microtransition densities converge geometrically to the critical point with λ(n)=n^σ, and relative variance Rv spikes flag them in advance.
- The trapping mechanism and cascade extend to Adam and AdamW under an explicit heavy-tailed noise model, conditional on checkable stopping times and a drift-margin condition.
- Empirically, DSI cascades match theory exactly in toy models (λ≈2.00) and appear with spectral-null support in grokking (λ=2.11, FPR=0.1%) and some UCI/vision settings, while German Credit serves as an honestly reported negative control.
Strengths
Careful discontinuity analysis
Corollary C.18 cleanly separates macroscopic (Θ(N)) from microscopic (O(1)) merge regimes, and Remark C.19 explicitly connects the latter to Riordan-Warnke (2011) proving Achlioptas processes continuous — rare honesty in the explosive-percolation-for-ML genre.
Honest negative control and null testing
German Credit (λ=1.99, R²=0.86) is reported with spectral-null FPR=80.2% as an explicit negative control (Table 1, §F.2); grokking uses a phase-randomized spectral null with FPR=0.1%.
Substantive Adam extension
Conditional rigor on the SGD side
Measured claims
Weaknesses
Theory-measurement gap
Statistically thin fits
Post-hoc interpretive freedom
Non-universal empirical coverage
Unverified load-bearing assumptions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This theory-heavy paper (arXiv 2609.02373, TU Munich) proposes that SGD's documented collapse toward symmetry-induced invariant sets should be read as a percolation process. The construction chain is: invariant sets from permutation symmetry (Defs. 2.1-2.2, adapted from Chen et al. 2023) → stochastic attractivity via drift dominance (Def. 2.4) → supermartingale trapping (Thm. 3.3, App. B) → attractor equivalence ∼A over subnetwork path measures (Def. 3.5) → a Reeb-graph quotient (Thm. 3.7) → temporal renormalization into a monotone percolation graph (Thm. 4.3, App. C.2). Central claims: (1) Sn symmetry forces simultaneous n-body block merges, giving order-parameter jumps ΔO=(n−1)i/N that are genuine discontinuities in the N→∞ limit only for Θ(N) components (Thm. 4.4, Cor. C.18); (2) microtransition densities obey p_c − p_{n·i} = n^(−σ)(p_c − p_i), a discrete scale invariance with λ(n)=n^σ (Thm. C.25); (3) relative variance Rv diverges at microtransitions and acts as a precursor (Thm. C.21); (4) trapping and the cascade survive Adam/AdamW under heavy-tailed noise with explicit stopping times (Thms. 5.4-5.5, App. D).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is discrete scale invariance (DSI) in this paper?
DSI means a system is self-similar only under a discrete set of magnification factors rather than any continuous rescaling. Here, because architectural permutation symmetry forces subnetworks to merge n-at-a-time in blocks, the densities at which successive merges occur form a geometric sequence converging to the critical point, with ratio λ(n)=n^σ.
Does the paper claim to explain grokking?
No. It reports that a 3-peak DSI variance cascade immediately precedes the grokking performance spike in a mod-17 transformer (λ=2.11, spectral-null FPR=0.1%) and explicitly states it does not claim this cascade is the sole driver of the transition.
Is the percolation transition in the paper discontinuous?
It depends on regime: the order-parameter jump ΔO=(n−1)i/N survives the infinite-network limit only when the merging components are already macroscopic (Θ(N)); for microscopic merges the jump vanishes and the transition becomes asymptotically continuous — consistent with the known continuity of Achlioptas processes (Riordan-Warnke 2011).
Is code available for the paper's experiments?
No code or data availability statement appears in the provided excerpt of the paper. The experiments are described as modest in scale (about one day on a single Tesla T4 GPU), and grokking hyperparameters (mod 17, 30,000 epochs, weight decay 0.5, learning rate 0.003) are given concretely.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: several middle sections (parts of Appendices C-F) were skipped by sampling, so judgments touching those gaps are hedged toward what the main text restates.
- Figures were assessed from captions and surrounding text only; the visual quality of the log-linear fits could not be verified from the extraction.
- The statement of Theorem F.1 itself fell in a sampling gap; its content (λ_eff = n·m^σ) is reconstructed from the shown derivation and Remark F.2.
- No code availability statement appears in the provided excerpt; one could in principle exist in a skipped section.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.