Grammatical "grandmother neurons" are rare in LLMs
cs.CL
Quick verdict
Introduces probe-free NSI across 68 minimal-pair paradigms and 7 LLMs; single-neuron grammatical selectivity is sparse and weak — well-controlled, credible.
Summary
The paper asks whether grammatical distinctions in LLMs are carried by individual 'grandmother' neurons or spread thinly across many weakly selective units. The authors introduce a probe-free Neuron Separability Index (NSI): for each of 68 minimal-pair paradigms (67 BLiMP plus one pooled COMPS paradigm, spanning four domains and 13 phenomena), they extract last-token activations for every residual-stream neuron in Qwen3-0.6B — replicating the analysis on Qwen3-1.7B/4B/8B, Pythia-410M, TinyLlama-1.1B and Llama-3.1-8B — compute a correlation-based separability between paired grammatical and ungrammatical response vectors, and normalize it against a 500-permutation within-pair label-swap null, with no trained classifier. Raw separability saturates earlier for morphology (about layer 8.4 at 90% of peak) and syntax (9.5) than for the syntax-semantics interface (10.3) and conceptual contrasts (19.0). After normalization, selectivity is sparse and weak: on average only 2.5-3.9% of units per paradigm show positive NSI, mean NSI among sensitive units is about 0.5, and only 3 of 68 paradigms contain any NSI>2 unit — exactly one neuron each out of 28,672. Selectivity dissociates from whole-vector probing (probe accuracy vs behavior r=-0.03; vs mean NSI r=-0.56) and from behavior; ablating the three NSI>2 units and their neighbors is no more damaging than 100 random same-layer ablations (p>=0.109). Limitations: last-token-only readout, no unit surviving BH-corrected testing, causal tests restricted to three paradigms in one model, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A probe-free, permutation-normalized Neuron Separability Index can quantify single-neuron grammatical selectivity uniformly across 68 minimal-pair paradigms and 7 LLM checkpoints.
- After permutation normalization, single-unit selectivity is sparse, weak and narrow: on average 2.5-3.9% of units per paradigm have positive NSI, and only 3/68 paradigms (Qwen3-0.6B) contain any NSI>2 unit — one neuron each out of 28,672.
- Raw separability saturates earlier for morphology (90% of peak at ~layer 8.4) and syntax (9.5) than for the syntax-semantics interface (10.3) and conceptual contrasts (19.0).
- Whole-vector linear decodability, single-neuron selectivity, and behavioral competence are empirically dissociated (probe vs behavior r=-0.03; probe vs mean NSI r=-0.56).
- Zero-ablating the three rare NSI>2 units plus their top same-layer neighbors is no more damaging than 100 random same-layer ablations (all p>=0.109), so activation selectivity does not imply causal reliance.
Strengths
Unprecedented coverage with a single uniform metric
One classifier-free index applied identically to all neurons across 68 paradigms (67 BLiMP + 1 pooled COMPS), 4 domains, 13 phenomena, and 7 checkpoints from 4 families (Section A claims first survey at this scale; Table 2 and Fig 7a-b show consistent 2.52-3.94% positive-NSI rates and 1.72-2.68 mean breadth across models).
The permutation null genuinely controls lexical confounds
Within-pair label swapping preserves lexical content while destroying the grammatical contrast (Section 2.2). Pairing controls (Table 4) show random good-good/bad-bad pairs give raw ~0.50 with zero NSI>2 units, while cross-item good-bad pairs manufacture 50 paradigms with outliers — direct evidence both item alignment and null normalization are necessary.
Critical-token deletion validates the estimand
Statistical honesty about the null's non-Gaussianity
The three-way dissociation is a real contribution
Weaknesses
No unit survives corrected significance testing
Single last-token, residual-coordinate readout
Causal coverage is thin and within-benchmark
Domain comparisons confounded by paradigm granularity
Unjustified probe layer and unexplained NSI>0 asymmetry
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses the granularity question in LLM interpretability: are grammatical distinctions localized in individual highly selective units ('grandmother neurons') or distributed across many weakly tuned ones? Three claims are advanced: (1) raw separability saturates earlier for morphology and syntax than for syntax-semantics interface and conceptual contrasts; (2) after permutation normalization, single-unit selectivity is sparse, weak and narrow — only 3 of 68 paradigms contain any NSI>2 unit in Qwen3-0.6B; (3) whole-vector decodability, single-neuron selectivity, and behavioral competence dissociate, and ablations separate selectivity from causal reliance. Coverage is broad: 68 paradigms (67 BLiMP + 1 pooled COMPS), 4 domains, 13 phenomena, ~116,300 templated pairs, and 7 checkpoints from 4 families. Scope is explicitly English, decoder-only, last-token readout.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are 'grandmother neurons' in the context of LLMs?
The term comes from neuroscience: single neurons that selectively encode one specific concept. Applied to language models, it asks whether a grammatical distinction (e.g., subject-verb agreement) is concentrated in a few highly selective artificial neurons or spread thinly across many weakly tuned ones.
What is the Neuron Separability Index (NSI)?
A probe-free metric that scores each artificial neuron on how consistently its activations differentiate grammatical from ungrammatical minimal-pair sentences. The raw score (based on correlation between paired response vectors) is normalized against a null distribution built by randomly swapping grammatical labels within pairs 500 times, so high NSI means selectivity beyond lexical overlap and noise; NSI>2 is treated as the strong-selectivity threshold.
Did the paper find grandmother neurons in LLMs?
Essentially no. In the main model (Qwen3-0.6B), only 3 of 68 linguistic paradigms contain even one neuron above the strong-selectivity threshold, each being a single unit out of 28,672; on average only 2.5-3.9% of neurons show any positive selectivity per paradigm, and ablating the rare strong units had no effect distinguishable from random ablations.
Which models and datasets were used?
Seven open-weight decoder-only checkpoints: Qwen3-0.6B/1.7B/4B/8B, Pythia-410M, TinyLlama-1.1B, and Llama-3.1-8B, evaluated on the BLiMP benchmark (67 paradigms) plus one pooled COMPS paradigm — about 116,300 templated English minimal pairs organized into 4 domains and 13 phenomena.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text omits several spans (end of Section 2.2, middle of Section 4.3, and the Appendix E probing setup); judgments rely on captions, tables, and remaining prose.
- Figures are available only as embedded text and captions; curve shapes and heatmap structure could not be independently verified.
- No code-availability statement is visible in the extracted text; its absence may be an extraction artifact rather than a fact about the paper.
- Exact per-paradigm item counts and the COMPS pooling procedure are described only at a high level (Appendix B), limiting precision of the granularity-confound assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.