The Attention Within: Consensus Dynamics in Selective State Space Models
Jo\~ao Pedro Silvestre, \'Alvaro Rodr\'iguez Abella, Paulo Tabuada
cs.LG · cs.AI · cs.SY · eess.SY · math.DS · math.OC
Quick verdict
ISS proofs show Mamba-2's recurrence drives token consensus as attention does, novel for time-varying weights, but theorems exclude the gate/SiLU.
Summary
The paper asks whether the recurrence at the core of selective state space models (SSMs) drives tokens to consensus the way attention does in transformers. The authors derive a continuous-time, closed-form token dynamics for Mamba-2 — capturing multi-dimensional tokens, time-varying per-layer weights, and RMSNorm-induced spherical projection — and analyze it on products of spheres using input-to-state stability (ISS) and the causal cascade structure of the recurrence. Under bounded weights, a unique dominant eigenvalue of the mixing matrix D_ij with a uniform spectral gap, and a persistency-of-excitation (PE) condition on the interaction kernel, they prove local exponential stability of the 2^ℓ sign-variant consensus equilibria (tokens at ±e_μ̃), claimed as the first such result for time-varying SSM weights. Strengthening PE to global positive definiteness yields an explicit domain-of-attraction boundary via an 'effective threshold' c_eff, with a possible positive-measure trapping set in the anisotropic case. Experiments on pretrained mamba2-130m (depth extended to 200 by cycling 24 layers, 50 tokens, 50 runs) show the recurrence clusters tokens, that the output gate limits the extent of consensus (λ̄1 ≈ 0.45 with gate vs ≈ 0.8 without), and that SiLU biases the interaction term positively, matching Lemma 4.1. Key limitation: the analyzed model excludes the gate and SiLU — precisely the components the experiments show govern actual behavior.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The Mamba-2 recurrence admits a closed-form input-output token update (Lemma 2.1) whose first-order approximation in τ yields a continuous-time dynamics on (S^{n-1})^ℓ capturing multi-dimensional tokens, time-varying weights, and RMSNorm.
- Under bounded weights, a unique dominant eigenvalue of D_ij with uniform spectral gap, and persistency of excitation of the interaction kernel, every sign-variant consensus equilibrium (all tokens at ±e_μ̃) is locally exponentially stable (Theorem 4.1), the first such result for time-varying SSM weights.
- Under global positive definiteness of S_BC(t), the domain of attraction is characterized by an effective threshold c_eff: initial conditions whose tokens stay above |z^μ̃_i| > c_eff converge to consensus, while anisotropy can create a positive-measure trapping set (Theorem 5.1).
- Experiments on pretrained mamba2-130m identify the output gate as the component limiting consensus (λ̄1 ≈0.45 with gate vs ≈0.8 without) and show SiLU biases the interaction term positively, consistent with Lemma 4.1's expectation calculation.
Strengths
First consensus results for selective SSMs with time-varying weights
Prior SSM token-dynamics analyses ([25] Vo et al., [26] Joseph et al.) are confined to time-invariant parameters without normalization; Theorems 4.1 and 5.1 handle per-layer time-varying weights, multi-dimensional tokens, and RMSNorm-induced spherical projection, transferring the cascade+ISS machinery the authors developed for autoregressive transformers [23],[24].
Complete, self-contained stability theory with explicit constants
The paper derives the input-output form (Lemma 2.1), reduces the analysis to scalar Lyapunov functions V_i = 1 − b_i, proves PE-based local exponential stability with explicit recursive rates (eqs. 13–14), and gives a quantitative domain-of-attraction threshold c_eff (Theorem 5.1); appendix proofs, including the discrete ISS Lemma 1.2, appear complete in the text.
Experiments calibrate the theory rather than merely illustrate it
Actionable architectural insight connecting to the over-smoothing literature
Weaknesses
Large theory–practice gap, acknowledged but not closed
PE verification is only along sampled trajectories
Strong and empirically unexamined structural assumption
Thin single-model experimental design without uncertainty reporting
Continuous-time approximation error never quantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do Mamba / state space models suffer from token uniformity (over-smoothing) like transformers?
According to this paper, yes at the mechanism level: the SSM recurrence provably drives tokens toward consensus — local exponential stability of equilibria where every token aligns with a dominant direction ±e_μ̃ — under a persistency-of-excitation condition on the interaction kernel. In depth-extended experiments on pretrained mamba2-130m, the recurrence alone raises the leading-direction energy λ̄1 to about 0.8, but Mamba's output gate limits it to about 0.45, so full over-smoothing is prevented mainly by gating.
What is the role of Mamba's output gate according to this paper?
The output gate is identified as the main component attenuating consensus. On mamba2-130m runs with depth extended to 200 layers, keeping the gate holds the leading-direction share λ̄1 near 0.45 and the effective number of token directions r near 4, whereas removing the gate lets λ̄1 rise to about 0.8 and r fall to about 1.4 — i.e., the gate is what stops tokens from collapsing onto a single direction.
What assumptions are needed for the consensus theorem to hold?
Three kinds: (1) learned weights stay bounded over time; (2) one channel μ̃ has the unique dominant eigenvalue of the mixing matrix D_ij for all times and states, with a uniform positive spectral gap; and (3) persistency of excitation — the interaction kernel must be positive on average over windows of length T near the consensus axis. A stronger pointwise positive-definiteness condition replaces (3) and yields an explicit domain-of-attraction threshold c_eff.
How does this differ from previous analyses of SSM token dynamics?
Earlier results (e.g., Vo et al., ICLR 2025; Joseph et al., ICLR 2025) analyzed SSMs with time-invariant parameters and without normalization. This paper is presented as the first consensus/stability analysis for selective SSMs with time-varying per-layer weights and with RMSNorm modeled as projection onto the unit sphere, which matches how trained Mamba-2 layers actually vary with depth.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures could not be visually analyzed; plateau values and trajectories are taken from in-text numbers and captions only.
- The text extraction appears essentially complete (including appendix proofs), but symbol layout in the PDF may have introduced minor artifacts, e.g., '̺' rendering.
- No code or data availability statement appears in the extracted text; its absence could be an extraction artifact rather than an omission by the authors.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.