Hallucination Neurons and Where to Find Them: An Investigation into the existence of Hallucination Neurons
Huseyin Cavus, Sebin Sabu, Joshua Spear, Jaskaran Singh Kawatra, Pavithra Rajendran
cs.AI
Quick verdict
Five-step diagnostic protocol shows H-neuron detection and causal effects replicate in Gemma models, but neuron selection is not uniquely localized.
Summary
This paper re-examines claims that small sets of feedforward "H-neurons" in LLMs detect and causally control hallucination. The authors propose a five-step diagnostic protocol — feature-correlation analysis, bootstrap stability, sparse-vs-dense ranking agreement, control-neuron intervention, and cross-dataset/model overlap — and apply it to the original H-neuron pipeline on Gemma 3 4B and MedGemma 4B over TriviaQA, BioASQ, and NQ-Open. Detection replicates: AUROC gaps are positive in every condition and strongest on BioASQ (+0.474 for Gemma 3 4B), and suppressing the selected neurons significantly lowers judge accuracy beyond random same-layer controls (n=500, five seeds; McNemar p<0.001 and p=0.026), though effects are modest (about 2.4 and 0.9 percentage points). Localization, however, is not supported: 19 of 22 selected neurons correlate above |r|=0.7 with non-selected features, bootstrap Jaccard reaches only 0.45-0.69, and L1/L2 rankings overlap by just 0-25%. Cross-dataset overlap is partial, with a few units such as (L16, N4146) recurring across dataset and model shifts. Limitations include evaluation of only the Gemma model family, ad hoc diagnostic thresholds, and no capability-preservation check after intervention.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sparse L1 probing over CETT features replicates H-neuron detection across Gemma 3 4B and MedGemma 4B on three QA datasets, with positive AUROC gaps up to +0.474 (BioASQ).
- Targeted suppression of the selected H-neurons produces statistically significant judge-accuracy drops beyond random same-layer neuron baselines (n=500, five seeds, McNemar tests).
- The identified neuron sets are not uniquely localized: they correlate highly with non-selected features, show only moderate bootstrap stability (Jaccard 0.45-0.69), and disagree severely with L2 rankings (0-25% overlap).
- Cross-dataset and cross-model neuron overlap is statistically significant but partial, with a few units (e.g., L16,N4146) recurring across dataset and model shifts, suggesting a partially shared mechanism.
- A five-step diagnostic protocol (D1-D5) should become a minimum validation standard distinguishing detection claims from localization claims in sparse-neuron interpretability.
Strengths
Clean separation of detection vs. localization claims
Sections 1 and 5.1 explicitly decouple 'the probe detects' from 'these specific neurons uniquely encode the behavior', grounding the critique in Lasso non-uniqueness under correlated features (Tibshirani 1996; Zou & Hastie 2005) — a statistically literate framing that most neuron-localization papers lack.
Replication is broad and strong
All six model-dataset conditions in Table 1 show positive AUROC gaps (+0.112 to +0.474), all nine zero-shot transfer directions in Table 5 are positive, and TriviaQA→BioASQ transfer (0.868) even exceeds TriviaQA's in-domain 0.811.
Causal validation with real controls
Honest, self-flagged limitations
Cheap, actionable protocol
Weaknesses
Metric-mismatch headline claim
Thin causal coverage
Cross-table numeric inconsistency
Narrow model scope
D5 null contradicted by D1
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a diagnostic replication study targeting the "H-Neurons" work (Gao et al., arXiv:2512.01797, ref [4]), which claims small sets of feedforward neurons in LLMs detect hallucination, causally control it, and originate in pre-training. The authors reframe the question: even if detection and population-level causal effects replicate, does the evidence support *unique localization* of the behavior to those specific neurons? They contribute (1) a five-step diagnostic protocol (D1-D5, Section 3.2) — feature correlation, bootstrap stability, sparse-vs-dense ranking agreement, control-neuron intervention, and cross-dataset/model overlap — proposed as a minimum standard for sparse-neuron localization claims; and (2) its application on Gemma 3 4B and MedGemma 4B across TriviaQA, BioASQ, and NQ-Open. Headline results: detection replicates everywhere (AUROC gaps +0.112 to +0.474, Table 1), suppression is statistically significant (Table 2), selection is demonstrably non-unique (Table 3), and cross-domain overlap is partial rather than universal (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are H-neurons in LLMs?
H-neurons are small sets of feedforward neurons that a sparse L1-regularized probe selects as detectors of hallucination. The term comes from a 2025 paper (Gao et al., arXiv:2512.01797) claiming these neurons both detect hallucinated behavior with high accuracy and causally control it when suppressed or amplified.
Does this paper confirm or refute the existence of hallucination neurons?
It refines the claim in both directions. Detection and population-level causal effects replicate: probes separate faithful from hallucinated responses (AUROC gaps up to +0.474 on BioASQ) and suppressing the neurons significantly lowers accuracy beyond random controls. But the paper shows the specific neurons are not uniquely localized — correlated alternatives exist, selections are unstable across bootstrap resamples, and neuron sets are only partially shared across datasets and models.
What is the five-step diagnostic protocol proposed in this paper?
It is a proposed minimum validation standard for sparse-neuron localization claims: (1) check whether selected neurons correlate strongly (|r|>0.7) with non-selected features, (2) measure stability under bootstrap resampling via Jaccard similarity, (3) compare L1-sparse and L2-dense feature rankings, (4) test interventions against random same-layer control neurons, and (5) measure neuron-set overlap across datasets and models with hypergeometric significance tests.
Is the code and data for this paper available?
The paper states that all experimental code, extracted features, neuron indices, and probes are available at github.com/huseyincavusbi/hprobes-protocol, using public datasets (TriviaQA, BioASQ, NQ-Open) and public models (Gemma 3 4B it, MedGemma 4B it). Note an internal inconsistency: Section 3.3 says artifacts are 'released upon acceptance' while the Data Availability section says they are already public.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based solely on the embedded extracted text; all six tables are present but no figures appear in the extraction, so any figure-only content could not be checked.
- The critique target (ref [4], Gao et al., arXiv:2512.01797) could not be consulted, so claims about what the original paper reported rest on this paper's description.
- Code availability could not be verified (no network access), and the paper itself contradicts on this point ('released upon acceptance' in Section 3.3 vs. 'publicly available' in Data Availability).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.