Correlated initialization of deep residual networks
Felix Benning, Ivan Nourdin, Giovanni Peccati
math.PR · cs.LG · stat.ML
Quick verdict
Settles Marion et al.'s correlated-init conjecture: scaling L^−H yields Hermite-driven Young limits; rigorous, blow-up unproven, experiments thin.
Summary
The paper analyzes residual networks whose output weights are correlated across layers at initialization, extending the independent-weight theory of Marion et al. (2025). Its main theorem (2.11) proves that when the weights are generated as v = φ(ξ) from a stationary Gaussian sequence with regularly varying correlation ρ(k) = k^−α ℓ(k) and a centered feature function of Hermite rank q, the critical scaling λ_L = L^−H ℓ(L)^{q/2} with H = 1 − αq/2 drives the interpolated hidden states, in Hölder topology, to the solution of a Young differential equation driven by a rank-q Hermite process (fractional Brownian motion when q = 1). This rigorously confirms the Marion et al. conjecture that correlated initializations interpolate between the independent-init Brownian SDE (λ ~ L^−1/2) and the perfectly correlated ODE (λ ~ L^−1). Sub-critical scaling provably collapses to the identity map (Corollary 2.12); super-critical blow-up remains conjectural (Remark 2.13). The proof builds new stability and Euler-convergence theory for Banach-space Young equations with a separate Hölder parameter path (Theorems 4.6, 4.8), of independent interest. MNIST experiments (depth 1000, width 30, Cauchy correlation, Hermite ranks 1 and 2) broadly support the predicted critical curve, but the observed transition slope matches neither that curve nor the maximal-update scaling, and the ReLU used experimentally violates the theory's boundedness assumption. Training-time claims are conjectural extrapolations of Chizat (2026). Code is public.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under correlated initialization v = φ(ξ) with regularly varying Gaussian correlation and Hermite rank q, the unique critical depth scaling is λ_L ~ L^−H ℓ(L)^{q/2} with H = 1 − αq/2, and at this scaling hidden states converge in Hölder topology to a Young differential equation driven by a rank-q Hermite process (Theorem 2.11).
- This rigorously resolves the Marion et al. (2025) conjecture that correlated initializations interpolate between the Brownian SDE of independent initialization (λ ~ L^−1/2) and the ODE of perfectly correlated initialization (λ ~ L^−1).
- Sub-critical scaling provably forces convergence to the identity map (Corollary 2.12), while super-critical blow-up remains conjectural (Remark 2.13).
- Correlation decay and Hermite rank are genuine initialization hyperparameters determining the infinite-depth dynamics, whereas iid initialization is universally Brownian up to normalization.
- New Banach-space stability and Euler-convergence results for parameter-dependent Young differential equations (Theorems 4.6, 4.8) are established and are of independent interest.
Strengths
Rigorous resolution of an open conjecture with genuine generality
Theorem 2.11 proves the Marion et al. [2025] conjecture for general regularly varying correlations ρ(k) = k^−α ℓ(k) plus nonlinear feature functions of arbitrary Hermite rank q, so the Hermite-process driver arises as a scaling limit rather than being built into the initialization as fBm increments, as in Marion et al.'s experiments.
New Young-equation toolkit proved in full
Section 4/5 deliver existence and uniqueness (Banach fixed point on shrinking intervals, glued via Lemma 5.5), local Lipschitz stability in initial condition, Hölder driver and separate parameter path (Theorem 4.6, Corollary 4.7), and Euler convergence with explicit rate |π|^{(1−α₁/α)(α+β−1)} (Theorem 4.8) via a discrete sewing lemma (Lemma 5.9); constant bookkeeping (Remarks 5.1–5.2) is unusually explicit.
Honest proven/conjectured separation
Candid theory-practice loop
Meaningful hyperparameter interpretation
Weaknesses
Experiments operate outside the proven regime
Half the phase transition is unproven
Thin experimental protocol
Headline theorem is not self-contained
Training-side analysis is extrapolation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies the large-depth (L → ∞) behavior of residual networks h_{l+1} = h_l + λ_L σ(w_l, h_l) v_l when the residual output weights v_l are correlated across layers at initialization. It targets the conjecture of Marion et al. [2025] that correlated initializations interpolate between two endpoints: independent weights at λ_L ~ L^−1/2 converge to a Brownian SDE; perfectly correlated (smooth/weight-tied) weights at λ_L ~ L^−1 converge to an ODE. The headline result, Theorem 2.11, proves that for v_i^l = φ(ξ_i^l) with ξ a stationary Gaussian sequence with regularly varying correlation ρ(k) = k^−α ℓ(k) and centered feature φ of Hermite rank q (assuming αq < 1 and v_i^l ∈ L^p for p > 2/(1−αq)), the unique critical scaling λ_L = L^−H ℓ(L)^{q/2}, H = 1 − αq/2, yields convergence in Hölder topology of the interpolated hidden states to the solution of a Young differential equation driven by a rank-q Hermite process; q = 1 is fractional Brownian motion, q = 2 the Rosenblatt process. Auxiliary claims: sub-critical scaling collapses to the identity (Corollary 2.12, proved); super-critical blow-up (Remark 2.13, conjectured); the Section 4 Young-equation machinery is offered as an independent contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the critical depth scaling for residual networks with correlated initialization?
According to this paper's Theorem 2.11, if layer-l output weights are generated from a stationary Gaussian process whose correlation decays like k^−α and the generating feature function has Hermite rank q, the critical scaling is λ_L ≈ L^−H with H = 1 − αq/2: exactly this scaling converges to a Hermite-process-driven differential equation, smaller scalings provably collapse to the identity map, and larger scalings are conjectured to blow up.
What is a Hermite process, and when does it reduce to fractional Brownian motion?
A Hermite process of rank q is a self-similar process with Hölder-continuous paths defined via a q-fold Wiener–Itô integral. For q = 1 it is exactly fractional Brownian motion with Hurst parameter H = 1 − αq/2; q = 2 gives the non-Gaussian Rosenblatt process.
Does this paper apply to ReLU networks?
Not directly. The main theorem assumes a bounded, locally Lipschitz activation (Assumption 2.5; satisfied by tanh, sigmoid, erf, arctan), whereas the authors' MNIST experiments use ReLU. The empirical critical-curve behavior under ReLU suggests the phenomenon extends beyond the proven regime, but this extension is not proved.
Is code available for the paper's experiments?
Yes. The experiments use a public fork of Marion et al.'s scaling-resnets code at github.com/FelixBenning/scaling-resnets (footnote 3), trained on MNIST with depth 1000, width 30, Cauchy correlation, and Hermite-rank-1 and rank-2 feature functions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 appear only as caption text in the extraction; the visual heat-map content of Figure 2 could not be inspected, so experimental claims rely on captions and surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.