Symmetry without a manifold: intrinsic dimension on orbits
Chon-Fai Kam, Miloud Bessafi, Fr\'ed\'eric Cadet
cs.LG · stat.ML
Quick verdict
Group-orbit data make intrinsic dimension undefined and modular-addition loss exponential in width; well-controlled, but one width decade and abelian only.
Summary
The paper asks what happens to geometric derivations of neural scaling laws when their data-manifold input does not exist. For modular addition in Z_p, the exact algebraic solution embeds tokens as an orbit of Z_p acting by isometries. The authors prove (Proposition 1) that on any such orbit the nearest-neighbour distance ratio underlying the Facco et al. two-NN intrinsic-dimension estimator is a point mass, so the estimator is undefined; the embeddings even have r1=r2 exactly (Lemma 6), and under epsilon-noise the recovered dimension tracks 1/epsilon with no plateau while a torus control stays near 2 (Lemma 8, Appendix B). Replacing the expected power law, held-out loss follows L = L_inf + A exp(-c h^alpha) in width h (R^2 0.982-0.995 versus 0.857-0.906 for a floor-matched power law, delta-AIC 25.5-40.9). Across eight group orders, the rate c moves 47x under weight decay but only ~1.10x across group order (within three-seed resolution), supporting a regularisation-budget account over capacity counting; critical width falls with p, opposite to per-irreducible-representation capacity predictions. The spectral mode count K_eff is shown to be a zero-parameter occupancy sampling statistic. Limitations: alpha is unidentified over barely one decade of usable width; the rate factorisation fails where data are scarce (quadratic at p=23, ReLU throughout), attributed to data supply via a training-fraction experiment; no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On any finite orbit of a group acting transitively by isometries, the two-nearest-neighbour ratio statistic is a point mass, so the Facco et al. intrinsic-dimension estimator is undefined, and under epsilon-perturbation its output tracks 1/epsilon with no scale-free plateau (Proposition 1, Lemmas 6 and 8).
- Held-out loss on Z_p modular addition is exponential in hidden width with a floor, L = L_inf + A exp(-c h^alpha), preferred to a floor-matched power law at every group order (R^2 0.982-0.995 vs 0.857-0.906; delta-AIC 25.5-40.9).
- In the data-sufficient regime the decay rate factorises into regularisation and group terms with the latter flat: weight decay moves c by a factor of 47 while group order moves it by ~1.10, below three-seed resolution, at every fixed alpha in [0.75, 2].
- The aggregate spectral participation ratio K_eff behaves as a zero-parameter occupancy sampling statistic (R^2=0.907) rather than a capacity variable (pooled R^2=0.590 vs 0.917 for width alone), so within-size capacity correlations are artefacts.
- Critical width falls with group order (34.1 at p=23 to 26.6 at p=113, slope -0.0738, R^2=0.934), contradicting capacity counting that assigns fixed neurons per irreducible representation; data supply (pairs growing as p^2/2 versus modes as p/2) explains the direction.
Strengths
Structural, provable degeneracy result rather than a mismeasurement claim
Proposition 1 proves via a one-line isometry argument that the two-NN ratio is a point mass on any transitive finite group orbit; Lemma 6 strengthens this for the Z_p embeddings to r1=r2 exactly (verified to 1e-14), and Lemma 7 identifies full-frequency content as a regular simplex with pairwise cosine -1/(p-1) at std 5e-16.
The perturbation analysis is quantitative and falsifiable
Lemma 8 predicts the perturbed estimate equals C/epsilon with an epsilon-independent constant; validated by C/s constant at 0.06494 across 8x embedding rescaling and d-hat*epsilon constant to ~1% over a 30x range of epsilon (Table 4), with a genuine torus control returning 1.98-2.20 over the same range.
Model comparison avoids the three-parameter fallacy
Experimental design built so competing accounts can disagree
Unusually candid protocol and failure documentation
Weaknesses
The stretched-exponent alpha is unidentified and the width range is under one decade
The group-order invariance claim is absence of evidence
The headline factorisation holds only in a narrow regime
Unresolved tension with the paper's own norm-budget calculation
No code or data release; single fixed data split
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'intrinsic dimension is undefined on group orbits' mean?
The popular two-nearest-neighbour intrinsic dimension estimator assumes data points are random samples from a density on a manifold. On a group orbit (e.g., the p token embeddings of an exact modular-addition solution, where a group acts transitively by isometries), every point has an identical multiset of distances to the others, so the ratio of second- to first-nearest-neighbour distances is the same everywhere and the estimator's regression has no signal. Adding noise yields a number, but it scales like 1/noise-level, tracking the probe scale rather than any geometry, whereas a genuine torus keeps returning about 2.
Does this paper disprove neural scaling laws?
No. It shows that a specific class of derivations - those that take a data-manifold intrinsic dimension as input - has no input on algebraic tasks like modular addition in Z_p, whose solutions are group orbits rather than sampled manifolds. On those tasks the loss is better described by an exponential in network width with a floor than by a power law. The claims are explicitly scoped to this regime; the authors note continuous attractors (e.g., tori) are not affected.
Why does the paper find exponential rather than power-law scaling with width?
Empirically, across eight group orders and fourteen widths, the floor-annotated exponential L = L_inf + A exp(-c h^alpha) fits held-out loss with R^2 between 0.982 and 0.995 versus 0.857 to 0.906 for a power law given the same floor and fitting protocol, with AIC preferring the exponential by 25 to 41 points. Supporting theory: the exact norm-constrained algebraic solution also yields an exponential in the number of Fourier modes carried, with an explicit closed-form Dirichlet-kernel calculation.
What role does weight decay play according to this paper?
In the regime with enough training data, the rate of the exponential is controlled by regularisation rather than task size: sweeping weight decay moves the rate by a factor of about 47, while sweeping group order from 47 to 113 moves it by only about 1.10 - within the experiment's seed-to-seed resolution. This supports a 'regularisation budget' account over 'capacity' accounts that tie performance to the number of irreducible representations.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions and quoted axis ranges in the extracted text; visual residual patterns and fit quality could not be inspected directly.
- The extraction appears complete (all appendices A-F present), but no supplementary material or code repository link is visible, so code availability is judged from absence of a statement.
- The review relies on the authors' reported R^2, AIC and seed statistics; fits were not independently reproduced.
- arXiv ID 2609.17926 and venue 'NeurReps 2026' are as stated on the preprint and were not externally verified per task constraints.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.