A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

Summary

The paper develops a statistical-mechanical description of neural-network training formulated directly in function space. For mean-squared loss, error dynamics are exactly ė=−Me with learning operator M=JJ*, and the multiplicity of parameter configurations realizing a function-space state contributes a density of states Ω whose log-curvature defines a statistical operator B. Combining a conditional Langevin weight (mobility K=M² under isotropic error-source noise) with Ω and integrating local Gaussian fluctuations on a finite-dimensional active sector yields the fluctuation free energy Φ_fluc=(σ²/2)logdet(M⁻¹+B). Theorems 4.1–4.2 prove this term is rotationally stationary if and only if [M,B]=0, is globally minimized on the fixed-spectrum orbit by reverse spectral pairing (large M eigenvalues with small B eigenvalues), and Corollary 4.3 gives a local restoring force there. For ReLU networks under assumed conditions R1–R3, B=σ²L*KL with L≈CD² the coarse-grained second-derivative operator, so low-B directions correspond — up to bounded anisotropy κ_K — to low coarse-grained structural curvature, predicting a bias toward faster relaxation along smooth, data-adaptive directions. The paper is honest that this is one conditional contribution among possibly many: the slow dynamics of M are not derived, the factorization A1 and ReLU conditions R1–R3 are assumed rather than derived, and no experiments, simulations, or computed examples of Ω or B for any concrete network are provided.