On the second-order optimization for spiking neural networks
cs.LG · cs.CV
Quick verdict
Proposes SpiKFAX, a Kronecker-factored Fisher optimizer for SNNs; careful derivation, consistent gains, but no second-order baselines and thin statistics.
Summary
The paper introduces SpiKFAX, a second-order optimizer for spiking neural networks (SNNs) built on a Kronecker-factored approximation of the Fisher information matrix. Starting from leaky integrate-and-fire neurons with surrogate gradients, the authors derive backpropagation-through-time recursions (Lemmas 1-2) and show that, under three explicitly stated assumptions - layer-wise block diagonality, independence of activations and deltas, and temporal homogeneity of input second moments - the Fisher of a time-shared weight factors as A⊗G, where A is the second moment of the spike-rate vector and G that of summed per-timestep deltas (Lemma 3). Theorem 1 quantifies cost: inverting the Kronecker factors scales as O(m³+n³) per layer instead of O(N³) for the full Fisher. Experiments use SNNTorch on five spiking architectures and seven datasets (MNIST, F-MNIST, CIFAR10, CIFAR100, N-MNIST, CIFAR10-DVS, DVS128 Gesture), five runs each, against SGD, Adam, and AdamW. SpiKFAX is best in every Table 1 cell - e.g., 99.22% on N-MNIST with S-VGG11, 76.13% on DVS128 with S-LeNet5, and 70.27% vs 49.24% for S-ResNet18 on DVS128 - at 1.2-1.4× per-iteration overhead. Main limitations: no comparison to the curvature-based methods it cites (K-FAC, AdaFisher), no significance testing despite large baseline variances (up to ±10), a 99.92% N-MNIST figure that does not match Table 1, missing hyperparameter details, and static-dataset results shown only as figures.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Spiking activations produce sharper loss landscapes than ANN counterparts, which diagonal-curvature optimizers like the Adam family fail to exploit (Fig. 1, Sec. 1).
- Under three stated assumptions, the Fisher information matrix of a time-shared SNN weight admits a Kronecker-factored approximation A⊗G built from spike rates and summed surrogate deltas (Lemma 3).
- Inverting the Kronecker factors costs O(m³+n³) per layer, drastically cheaper than the O(N³) inversion of the full Fisher (Theorem 1).
- SpiKFAX achieves the best test accuracy in every reported comparison cell across five architectures and seven datasets, with only 1.2-1.4× per-iteration overhead (Table 1, Figs. 2-4).
Strengths
Careful, layered derivation
Sec. 2 states Assumptions 1-3 separately and the Lemma 3 proof marks exactly where each enters; Lemma 1's recurrent Jacobian β − Vthr σ′ correctly reflects the subtractive reset, and Remark 1 honestly flags the extra KFC approximation needed for conv layers.
Honest framing of Fisher variants
Definition 1 explicitly distinguishes the true Fisher from the empirical Fisher and notes the latter does not converge to the former away from zero-residual optima — a subtlety many K-FAC-style papers omit.
Quantified tractability claim
Broad and consistent evaluation
Practical overhead
Weaknesses
No curvature-based baselines
Statistical reporting gaps
Internal numeric inconsistency
Reproducibility details missing
Half the evidence is graphical only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This conference-length paper introduces SpiKFAX, a second-order optimizer for spiking neural networks (SNNs) that preconditions gradients with a Kronecker-factored approximation of the Fisher information matrix, adapted to time-recurrent, surrogate-gradient dynamics. The authors claim three contributions (Sec. 1): the optimizer itself, a computable Fisher approximation for SNNs, and experiments across five architectures and seven datasets showing consistent accuracy and stability gains over SGD, Adam, and AdamW. The motivating premise is Fig. 1: the loss landscape of a spiking VGG11 is visibly sharper than its ANN counterpart, and diagonal (Adam-family) curvature cannot capture off-diagonal structure in such landscapes. The work sits in the K-FAC lineage (Martens & Grosse [5]) and claims to be the first Kronecker-factored Fisher treatment designed around SNN dynamics.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SpiKFAX?
SpiKFAX is a second-order optimization method for spiking neural networks that approximates the Fisher information matrix as a Kronecker product A⊗G — the input spike-rate second moment times the second moment of summed per-timestep gradient deltas — and uses its inverse to precondition gradients during backpropagation-through-time training.
How much slower is SpiKFAX than Adam?
Per iteration it is about 1.2x slower than Adam and AdamW and 1.4x slower than SGD on the DVS128 Gesture benchmark, while inverting the full Fisher matrix exactly would be computationally intractable.
What accuracy does SpiKFAX achieve?
In the paper's Table 1 it is the best method in every tested cell, e.g. 99.22% on N-MNIST with a spiking VGG11, 76.13% on DVS128 Gesture with spiking LeNet5, and 70.27% versus 49.24% for the best baseline with spiking ResNet18 on DVS128 Gesture.
Is the SpiKFAX code available?
The paper states the code is available in the authors' repository, but the extracted text contains no URL, and key hyperparameters (inversion interval, damping, timestep count, surrogate function) are not reported, so independent reproduction may require contacting the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 are curve plots; all static-dataset (MNIST, F-MNIST, CIFAR10, CIFAR100) results exist only as figures and could not be checked numerically.
- Table 1 digits are concatenated in the extracted text (e.g., 92.661.27 read as 92.66±1.27); parsing is confident but a digit-level misassignment cannot be fully excluded.
- The 99.92% N-MNIST figure in Sec. 3 does not match Table 1's maximum (99.22±0.01); it may refer to the S-MLP architecture, which is missing from the extracted table.
- Repository URL, hyperparameter values (Tinv, damping, τ, surrogate), and any appendix are absent from the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.