NGN: Learning Neural Network Size as a Differentiable Count
cs.LG
Quick verdict
NGN learns model size as a differentiable boundary; broad, honest experiments, but size choice shifts to tuning λ and PEFT results are single-seed.
Summary
The paper introduces the Neurogenesis Network (NGN), a parameterization that turns neural network size into a differentiable count. Each ordered group of structural components (MLP blocks, conv branches, attention heads, FFN slices, Mamba state dimensions, LoRA ranks, adapter widths) receives a single learnable boundary t; sigmoid gates g_k(t)=sigmoid(β(t−k+0.5)) implement a soft prefix, and the objective adds a linear capacity price λt plus a periodic α·sin²(πt) term that snaps t to integers, enabling deployment by truncating at round(t). Controlled 1D regression studies isolate each schedule component: removing the snapping term widens the soft–truncated gap from 5.4e-5 to 4.8e-4, and fixed sharp β selects too-small counts. A VAE on manifolds of known intrinsic dimension recovers the true bottleneck in 49/50 seed–dimension cases. Across MNIST, CIFAR-10, Cora, TinyStories, depth learning, and Llama-3-8B PEFT/pruning, truncated prefixes usually cost little versus soft models and matched fixed retraining. Critically, the paper concedes fixed-size models are typically slightly better (Figure 3), optimization is slower, and λ must still be calibrated — the benefit is automatic size selection, not better models. Main limitations: headline PEFT results are single-seed per configuration with low integer-snap rates (20–34% on several tasks), TinyStories rows are single-seed, and the AWNN baseline is a controlled reimplementation that underperforms. Overall a carefully hedged method paper whose practical advantage over a small fixed-size sweep remains unquantified.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single learnable boundary per ordered component group makes structural count a differentiable quantity trainable jointly with weights, with λt supplying capacity pressure and α·sin²(πt) snapping the boundary to an integer for hard-prefix deployment (Eqs. 3–5).
- The learned count tracks known structural targets: 49/50 seed–dimension manifold VAE cases recover the true intrinsic bottleneck (§4.4).
- Truncated NGN prefixes usually perform close to soft models and to fixed models retrained at matched size, across seven-plus architecture families (Tables 3 and 5).
- Fixed-size models are typically slightly better at matched size and NGN optimizes more slowly, so the benefit is automatic size selection rather than better or faster models (§4.2, §6).
Strengths
Controlled analyses genuinely isolate the mechanism
Table 1's schedule ablations cleanly separate roles: removing α widens the soft/truncated gap from 5.43e-5 to 4.83e-4; fixed β=15 collapses the count to t=2.00 with 7.54e-4 fit; Figure 5 (50 seeds per initialization) shows geometric weights plus delayed α are jointly necessary for bidirectional grow/prune convergence.
Ground-truth validation of a learned structural quantity
The §4.4 manifold VAE experiment tests whether the learned count tracks a known target: 49/50 seed–dimension cases recover the true intrinsic bottleneck dimension (Tables 2/8), a rare direct check that the boundary means what it claims.
Unusual candor about the negative controls
Breadth of transfer with a single primitive
Reporting hygiene
Weaknesses
Single-seed results exactly where they matter most
λ-circularity undermines the no-sweep selling point
Baseline coverage gaps
Growth ceiling from exponential ordering
Fragile statistics in the controlled studies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes the Neurogenesis Network (NGN): one learnable scalar boundary per ordered group of structural components, gated by sigmoids (Eq. 3, g_k(t)=sigmoid(β(t−k+0.5))), trained with task loss plus a linear capacity price λt and an integer-snapping α·sin²(πt) term (Eq. 5), then deployed by truncating at round(t). Claims: (1) count is learnable jointly with weights; (2) learned counts track known targets — 49/50 manifold seed–dimension cases recover the true intrinsic bottleneck (§4.4); (3) truncation and matched fixed retraining cost little (§5); (4) one mechanism spans MLPs, CNNs, GCNs, Transformers, Mamba, LoRA, adapters, and pruning budgets (§5, App. G). The paper polices its own claims: §5.1 concedes reliable prune-from-high recovery is not observed across the full set, and §6 admits fixed models are typically slightly better.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is NGN in one sentence?
NGN (Neurogenesis Network) is a training method that treats a neural network's size — such as the number of blocks, attention heads, or LoRA ranks — as a single learnable scalar boundary that grows or shrinks during training and snaps to an integer, so the finished model can be truncated to that size for deployment.
Does NGN produce better models than simply training a fixed-size network?
No. The paper's own capacity-matched comparison (Figure 3) shows fixed models trained from scratch at the same size are typically slightly better, and NGN takes more optimization steps. Its stated benefit is automating the size choice, though the capacity price λ must still be tuned, so some of the sweep cost is relocated rather than eliminated.
What tasks and architectures were tested?
Controlled 1D regression and manifold autoencoding with known intrinsic dimension, MNIST and CIFAR-10 image classification, Cora graph classification, TinyStories language modeling with Transformers and Mamba, synthetic depth learning, and LoRA/adapter/pruning experiments on Llama-3-8B-Instruct and Pythia-410M.
Is code available?
The paper states that code and configuration files will be included in the supplementary material, and appendix tables list per-experiment structural hyperparameters; however, optimizer and learning-rate details are explicitly omitted, which slightly limits exact reproduction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 are referenced with numeric captions but not visually analyzable from the extracted text; judgments rely on the quoted numbers in captions and tables.
- Appendix C explicitly omits optimizer and learning-rate details, limiting a full reproduction audit.
- arXiv v1 preprint with unclear peer-review status; the cited AWNN (ICLR 2026) reference could not be verified from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.