Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence
Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui, Hinrich Sch\"utze
cs.LG · cs.CL
Quick verdict
Weight-cosine IO taxonomy finds consistent early-strengthen/late-weaken patterns in LLMs; rare weakening neurons have big impact — solid, mechanisms open.
Summary
The paper introduces an input-output (IO) analysis of GLU-based MLP neurons in transformer LLMs: for each neuron, cosine similarities among its input (win), gate (wgate), and output (wout) weight vectors define a taxonomy — strengthening, weakening, conditional variants, proportional change, orthogonal output, plus 'atypical' mismatch classes at threshold τ=0.5. A custom RefactorGLU preprocessing (App. D) exploits the sign symmetry of gated neurons so that behaviorally equivalent neurons receive identical classifications; significance is assessed against Gaussian and layer-specific mismatched-cosine nulls. Across 12 GLU LLMs (0.5B–9B) and 24 non-GLU models, patterns are consistent: conditional strengthening neurons dominate early-middle layers while weakening neurons concentrate in late layers (Figs. 2–3). On 20M Dolma tokens, cos(win,wout) correlates negatively with gate-positive frequency (Table 1: -0.91 for OLMo-7B vs. up to -0.69 in prior GELU-model work). Mean-ablating the 526 weakening neurons of OLMo-7B disproportionately shifts entropy, loss, and attribute rate relative to same-layer random baselines; conditional ablations attribute much of the effect to negative-gate quadrants (ii)/(iii), yielding first evidence (concurrent with Kong et al., 2026) that negative Swish gates are functionally load-bearing. Case studies (Sec. 8, App. I) probe single neurons. Main limitations: mechanisms remain speculative (the confidence-management hypothesis is unconfirmed and excludes late layers); multi-layer conditional ablations are brittle to sign flips; Dolma may mismatch closed models' training distributions; no MoE or >9B models.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across 12 GLU LLMs, read-write cosine structure is consistent: conditional strengthening neurons dominate early-middle layers while weakening neurons concentrate in late layers.
- IO cosine similarity strongly predicts activation frequency, with negative correlations up to -0.91 (OLMo-7B), stronger than prior GELU-model findings.
- The small weakening-neuron class (526 neurons in OLMo-7B) exerts outsize influence on entropy, loss, and attribute rate under ablation, exceeding same-layer random baselines.
- Much of this influence comes from negative-gate activations (quadrants ii/iii), showing negative Swish gate values are functionally meaningful, not merely a training-dynamics artifact.
Strengths
Simple, cheap, uniformly applicable method
Weight cosines take under a minute per model (App. A.1) and need no activations or training; applying it identically to 24 non-GLU and 12 GLU models yields strikingly consistent cross-model patterns (Fig. 2, Fig. 26), including replicating and strengthening Gurnee et al. (2024)'s frequency-cosine correlation in the gated setting (-0.91 vs. up to -0.69).
Careful null baselines for significance
Two baselines (i.i.d. Gaussian and layer-specific mismatched cosines, Sec. 4.3/F) explicitly rule out outlier-dimension artifacts; 95% randomness regions are drawn per cosine in Fig. 4 and validated empirically on a randomly initialized OLMo checkpoint (Fig. 8), so 'input manipulator' claims rest on more than eyeballing.
Novel and well-isolated finding on negative gate values
Principled, behavior-preserving preprocessing
Strong reproducibility and candor
Weaknesses
Statistical reporting is thin
Headline pooled correlation risks layer confounding
Equal-count ablation design does not isolate per-neuron impact
Causal story remains incomplete
Distribution mismatch in corpus-based experiments
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'weakening neuron' in a language model?
It is an MLP neuron whose input (reading) weight vector points in nearly the opposite direction of its output (writing) weight vector, i.e., a strongly negative cosine similarity between them. Such a neuron detects a direction in the residual stream and writes back that direction's opposite, weakening it. The paper finds these neurons mostly in late layers, and shows that although they are few (e.g., 526 in OLMo-7B), they activate often and have a disproportionately large effect on model behavior under ablation.
Do negative gate values matter in SwiGLU models?
They were commonly assumed to matter only for training dynamics, not for the trained model's function. This paper shows otherwise: ablating only the negative-gate activations of weakening neurons substantially changes output entropy and loss, because negative gate values flip the neuron's usual weakening behavior into strengthening-like behavior. The authors conclude that for mechanistic interpretability, Swish cannot be treated as equivalent to ReLU.
Which models did the paper analyze?
Twelve GLU-based LLMs (Llama-2-7B, Llama-3.1-8B, Llama-3.2-1B/3B, Gemma-2-2B/9B, Mistral-7B, Qwen2.5-0.5B/7B, OLMo-1B/7B, Yi-6B) using SwiGLU or GEGLU, plus 24 non-GLU models including the GPT-2 family, Pythia, OPT, BERT, T5, GPT-J, and Othello-GPT.
Is the code and data for this paper available?
Yes. The weight-analysis code is at github.com/sjgerstner/RW_functionalities, activation-based case-study tooling and visualizations at github.com/sjgerstner/gluscope and gluscope.github.io, and the regenerated subject-attribute dataset used for ablation metrics is also published. The authors state all main experiments run on a single RTX A6000 (48GB).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as textual descriptions in the extracted text; heatmap densities and bar heights (Figs. 2-6) could not be verified visually.
- Several replication claims (e.g., Llama-3.2-3B ablation results 'similar' to OLMo, other-model plots in supplementary material) are asserted but not shown in the provided text.
- Inline math and figure captions are partially garbled by extraction (e.g., weight names in Fig. 1); terminology was reconstructed from surrounding body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.