What Does Layer-Importance Reveal About Transformers and State-Space Models?
Istabrak Abbes, Nizar Islah, Irina Rish, Sarath Chandar
cs.LG · cs.AI
Quick verdict
NPA diagnostic links layer necessity vs plasticity: splits transformers from SSMs; rigorous controls, but causal-validation near chance and split fades at 32B+.
Summary
The paper asks whether layer-importance analysis built for transformers transfers to state-space models. It decomposes layer importance into Necessity (loss increase when a layer is bypassed) and Plasticity (magnitude of task-specific LoRA updates), and introduces Necessity-Plasticity Alignment (NPA), the mean Spearman correlation between the two rankings across tasks. Across roughly 20 public checkpoints (130M-70B) and ten tasks, NPA cleanly separates the families at small-to-mid scale: all evaluated transformers up to 14B parameters are negative (family mean -0.39), with necessity concentrating early (layer 0 alone carries ~62% of ablation mass) and plasticity late; Mamba-style SSMs are positive (mean +0.24), with overlapping profiles. RWKV is strongly positive (+0.54); the hybrid Zamba2 (+0.05) and Falcon-Mamba-7B (-0.02) sit near zero. Controls include adapter-free full fine-tuning on 13 checkpoints, update-norm normalization, and layer-0 removal. A linearized theory with explicit assumptions explains the transformer split via residual geometry and predicts its absence in SSMs. In a two-task continual probe, concentrating updates on the most plastic layers increases catastrophic forgetting in transformers but not in SSMs. Limitations: NPA collapses toward zero for Qwen3-32B and Llama-3.1-70B; causal-validation support rates are modest (LoRA-delta about 48-49%, with random baselines up to 63±21% on Qwen3-8B); the continual-learning probe is small-scale; theory assumptions are only partially verified empirically. Code and per-layer scores are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Necessity-Plasticity Alignment (mean cross-task Spearman between ablation-Necessity and LoRA-delta-Plasticity rankings) is negative for every evaluated transformer up to 14B parameters (family mean -0.39) and positive for the evaluated Mamba-style SSMs (mean +0.24).
- The sign split is not a LoRA artifact: an adapter-free full-fine-tuning control on 13 checkpoints preserves the sign in all cases, as do update-norm normalization and layer-0 removal.
- In transformers, ablation Necessity mass concentrates early (~62% on layer 0) while Plasticity concentrates late; in Mamba-style SSMs the two profiles overlap — explained by a residual-geometry theory with explicit assumptions A1-A4.
- NPA sign predicts continual-learning behavior: concentrating updates on the most plastic layers increases catastrophic forgetting in transformers but not in Mamba-style SSMs.
- The largest transformers evaluated (Qwen3-32B, Llama-3.1-70B) collapse toward zero NPA rather than entering the SSM regime.
Strengths
Clean conceptual decomposition with an estimator-agnostic diagnostic
Section 3 separates Necessity (bypass-ablation loss increase, skip path preserved) from Plasticity (per-layer LoRA update Frobenius norm) and defines NPA as their mean cross-task Spearman correlation, letting the pruning/interpretability and selective-PEFT literatures be compared on one plane (Fig. 1).
Unusually broad checkpoint coverage plus strong controls
~20 public checkpoints from 130M to 70B; the LoRA objection is pre-empted by an adapter-free full-FT control on 13 checkpoints where the adapter-free NPA analogue keeps the same sign in 13/13 cases (Appendix E.3), plus update-norm normalization and a layer-0-removal control (Sec. 5.3).
Falsifiable theory tied to measurable predictions
Elegant within-model identification via the hybrid checkpoint
Honest failure reporting and released artifacts
Weaknesses
Causal-validation utility is near chance
The dichotomy is scale-scoped and the collapse is explained post-hoc
Boundary taxonomy is partly definitional
Continual-learning probe is small
Theory assumptions are empirical premises, verified only on evaluated models
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a transfer question: does the layer-importance toolkit calibrated on residual transformers (pruning, selective PEFT, interpretability) measure the same thing on selective state-space models? It splits 'importance' into **Necessity** (per-task loss increase when a layer is bypassed, skip path preserved) and **Plasticity** (Frobenius norm of the per-layer LoRA update after task fine-tuning), and defines **NPA** — the mean Spearman correlation between the two layer rankings across ten tasks (HumanEval, MBPP, LongBench, SQuAD, GSM8K, MMLU, GPQA, ARC, PIQA, WinoGrande). Headline claims: (1) every evaluated transformer up to 14B parameters has negative NPA (family mean -0.39); (2) the evaluated Mamba-style SSMs are positive (+0.24); (3) the split survives an adapter-free full-FT control on 13 checkpoints, update-norm normalization, and layer-0 removal; (4) NPA sign predicts tier-dependent forgetting in a two-task continual probe; (5) Qwen3-32B and Llama-3.1-70B collapse toward zero rather than turning SSM-like.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Necessity-Plasticity Alignment (NPA)?
NPA is a diagnostic defined as the mean Spearman correlation, across tasks, between a layer's Necessity ranking (how much the loss increases when that layer is bypassed) and its Plasticity ranking (how large its LoRA weight updates are during fine-tuning). Positive NPA means the same layers are both necessary and plastic; negative NPA means pretrained dependence and adaptation concentrate at opposite ends of the network; values near zero mean no stable depth ordering exists.
Do layer-importance findings from transformers transfer to Mamba and other SSMs?
According to this paper, not directly: transformers up to 14B parameters show negative NPA (necessity early, plasticity late; family mean -0.39), while Mamba-style SSMs show positive NPA (overlapping regions; mean +0.24). However, the largest transformers tested (Qwen3-32B, Llama-3.1-70B) move toward zero, and boundary models such as Falcon-Mamba-7B (-0.02) and the hybrid Zamba2 (+0.05) sit near zero.
Which layers should you fine-tune to avoid catastrophic forgetting?
The paper's two-task probe suggests that in transformers, concentrating updates on the most plastic (usually late) layers increases forgetting, whereas in Mamba-style SSMs the placement effect disappears. The authors caution that their causal-validation gains over random placement are modest (LoRA-delta support around 48-49%, with a random baseline reaching up to 63±21% on Qwen3-8B), so the practical guidance should be treated as preliminary.
Is the transformer-vs-SSM split just an artifact of using LoRA?
The authors report an adapter-free control on 13 checkpoints where Plasticity is recomputed from full-parameter fine-tuning: full-FT and LoRA per-layer update profiles are positively rank-correlated on every checkpoint (from 0.96 on Qwen3-0.6B to 0.39 on Qwen3-14B), and the sign of the NPA analogue matches the LoRA-based sign in all 13 cases, so the split does not depend on low-rank adaptation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction omits or truncates several sections (parts of 4.2/5.4, the continual-learning probe protocol of Section 6, and portions of Appendices A and D), so those parts are assessed only via surrounding text and figure fragments.
- Figures are available only as text dumps; bootstrap CI widths in Fig. 1 and several Fig. 2 heatmap cells could not be fully verified against task labels due to extraction order.
- Appendix material referenced but only partially included (Lemma A.4, Theorems A.6, assumptions B1'-B2) means theoretical rigor is judged on the visible proof sketches.
- Per-task NPA values for Zamba2 and Falcon-Mamba are visibly mixed-sign in the extracted Fig. 2 rows, but exact task-to-value alignment could not be confirmed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.