Large Language Models with At Most One Spike per Neuron
Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud
cs.NE · cs.CL
Quick verdict
Claims first end-to-end TTFS spiking LLM (BERT, GPT-2 up to 1.5B): GLUE/commonsense near ANN, perplexity lags; softmax, stats, code are thin.
Summary
The paper introduces R-TTFS, a reference-based variant of time-to-first-spike (TTFS) coding that places the reference time at the midpoint of the timing window, enabling signed activations and an exact mapping to the linear regime of shift-ReLU (Sec. 4.1). Building on this, the authors encode the four LLM components that conventional TTFS cannot handle — embedding, LayerNorm (unrolled into seven spike-based steps, including a frozen two-layer approximation of 1/√x), attention, and dropout — and train the resulting networks end-to-end from scratch. On GLUE, TTFS-BERT-Base averages 80.5 versus 84.0 for standard BERT-Base, and TTFS-BERT-Large reaches 82.3 versus 85.3, clearly beating SpikeBERT (65.1) and approaching SpikeLM (81.3). For GPT-2, the Small model slightly exceeds its ANN baseline on commonsense reasoning (42.5 vs 40.7) while the 1.5B XL model trails (49.5 vs 51.6); language modeling clearly degrades (WikiText perplexity 20.4→26.6, LAMBADA accuracy 51.2→39.4 for XL), which the authors honestly flag as the main open problem. Energy is reported only as a spike-count proxy (0.80 Tr per neuron per inference vs 1.2 Tr for SpikeLM), with no neuromorphic hardware measurement. Main weaknesses: no seeds or variance anywhere, softmax and dynamic-weight mechanisms barely specified, and no code release mentioned.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A reference-based TTFS neuron (R-TTFS) with the reference at the timing-window midpoint enables signed activations while preserving an exact mapping to shift-ReLU linear layers (Sec. 4.1, Eqs. 3-4).
- All four LLM components lacking conventional TTFS mappings — embedding, LayerNorm, attention, dropout — can be represented or approximated in TTFS, permitting end-to-end trained spiking LLMs (Sec. 4.2).
- TTFS-BERT achieves GLUE averages of 80.5 (Base) and 82.3 (Large) versus 84.0/85.3 for ANN BERT, outperforming SpikeBERT (65.1) and approaching SpikeLM (81.3) (Table 1).
- TTFS-GPT-2 XL is the first spiking LLM scaled to 1.5B parameters with TTFS coding; commonsense reasoning is near the ANN baseline while language-modeling perplexity clearly degrades (Table 2).
- Under a spike-count proxy cost model, TTFS-BERT costs 0.80 Tr per neuron per inference versus 1.2 Tr (SpikeLM) and 3.2 Tr (SpikingBERT), explicitly not a hardware measurement (Sec. 6.1, Table 3).
Strengths
First complete TTFS formulation of the full transformer component set
Sec. 4.2 extends exact-mapping TTFS beyond MLPs to embedding (Sec. 4.2.1), a seven-step spike-based LayerNorm (Sec. 4.2.2, Eqs. 6-16), attention (Sec. 4.2.3), and dropout — enabling end-to-end training rather than conversion, which the authors argue avoids the approximation errors of Jiang et al. 2024 and Zhao et al. 2025.
Unusually honest claim scoping
The abstract, Sec. 5.2, and Sec. 6.1 explicitly disclaim parity on perplexity ('we do not claim parity'), label the energy figure a 'spike-count proxy... rather than a measurement', and the conclusion lists the LAMBADA gap and lack of hardware measurements as open limitations — rare candor in this subfield.
Scale demonstration with competitive numbers
Transparent spike-budget accounting
Weaknesses
No statistical uncertainty anywhere
Softmax and dynamic synaptic weights are underspecified
Training method through spike times not described
Notation errors undermine confidence in the derivations
No code or artifacts released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper builds spiking LLMs entirely on time-to-first-spike (TTFS) coding, where each neuron emits at most one spike per window. The core methodological claim is that a reference-based variant (R-TTFS, Sec. 4.1) with the reference at the timing-window midpoint restores signed activations and preserves the exact ReLU mapping of Stanojevic et al. within the linear regime of a shift-ReLU (Eqs. 3-4). The second claim is constructive: embedding, LayerNorm, attention, and dropout (Sec. 4.2) can each be expressed or approximated in TTFS, enabling end-to-end training of BERT and GPT-2 models up to 1.5B parameters — claimed as the first TTFS spiking LLM at that scale. The empirical claims are deliberately hedged: 'comparable' on NLU/commonsense, explicitly not comparable on perplexity ('we do not claim parity', Sec. 5.2), and energy reported only as a spike-count proxy (0.80 Tr/neuron vs 1.2 Tr SpikeLM, 3.2 Tr SpikingBERT; Sec. 6.1, Table 3). The abstract's candor about the proxy nature of the energy estimate is commendable and rare in this subfield.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TTFS coding in spiking neural networks?
Time-to-first-spike (TTFS) coding represents information by the precise timing of a neuron's first and only spike within a time window, instead of a firing rate. This paper's variant, R-TTFS, places a reference time at the window's midpoint so that both positive and negative values can be encoded as signed timing shifts.
Did the authors run their spiking language model on neuromorphic hardware?
No. All training and evaluation were done on GPUs (eight NVIDIA H100s). The energy analysis is a spike-count-based proxy estimate under an existing cost model, and the authors explicitly state that no neuromorphic hardware was measured; validating on a physical chip is left as future work.
How close is TTFS-BERT to normal BERT on GLUE?
TTFS-BERT-Base averages 80.5 versus 84.0 for standard BERT-Base, and TTFS-BERT-Large averages 82.3 versus 85.3 for BERT-Large. It clearly beats the spiking baseline SpikeBERT (65.1) and comes close to SpikeLM (81.3), but does not fully match the non-spiking models.
What is the main limitation of this spiking GPT-2?
Language modeling quality. Commonsense-reasoning accuracy stays close to the standard GPT-2 baseline, but perplexity clearly degrades: on the 1.5B XL model, WikiText perplexity rises from 20.4 to 26.6 and LAMBADA accuracy falls from 51.2 to 39.4. The authors identify closing this gap as the main open problem.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Mathematical notation is flattened by PDF extraction (sub/superscripts, fractions across Eqs. 1-23; e.g., Eq. 9 renders as '1 = Wj, 1 = Wμ'), so the derivations could not be fully re-verified and may differ typographically in the PDF.
- Figures (Fig. 1-9) are absent from the extracted text; statements relying on them (e.g., the quality of the 1/√x fit in Fig. 5, training curves in Figs. 6-7) rest on captions and surrounding prose only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.