Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
cs.CL
Quick verdict
Claims transformers and SSMs diverge in global representation geometry yet converge locally; solid descriptive study, though evidence is correlational.
Summary
The paper asks whether transformers and state-space models (SSMs) build similar internal representations despite fundamentally different architectures. Across five open models (Pythia-1.4B/2.8B, Falcon-7B, Mamba-1.4B/2.8B, FalconMamba-7B, and hybrid Jamba2-3B), the authors analyze residual-stream representations on roughly 500K FineWeb-Edu tokens and LAMA TREx factual relations. Globally, SSMs are far more isotropic with higher RankMe effective dimensionality, while transformer representations are dominated by a first principal component; in hybrid Jamba2, attention layers push representations toward anisotropy and subsequent Mamba layers reverse it, implicating attention as the collapse driver. Functionally, rank-constrained autoencoders and probes yield closely matched capacity and concept-decoding curves across architectures (Pearson r>0.84 between per-layer KL degradation curves), and PCA ablations show the transformer's dominant direction carries no extra conceptual information. Locally, ZCA-whitened CKA reveals high alignment of concept manifolds (mean-max roughly 0.86-0.87 for trained pairs on LAMA TREx versus 0.44-0.57 on general text), and mutual k-NN overlap is high at corresponding depths. Checkpoint analysis of Pythia shows local alignment emerges within about 4% of training while global geometry stabilizes near 25%. Limitations: probing evidence is correlational rather than causal, training dynamics rest on Pythia alone, and most comparisons lack significance testing.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SSM residual streams are markedly more isotropic with higher effective rank (RankMe) than transformer counterparts, and in hybrid Jamba2 attention layers induce anisotropy that subsequent Mamba layers reverse.
- Despite divergent global geometry, transformers and SSMs have tightly matched effective representational capacity, with near-identical degradation under rank-constrained autoencoders (Pearson r>0.84 across bottleneck ranks 64-512).
- Both architectures encode LAMA TREx relational facts in subspaces of similar dimensionality, and the transformer's dominant PC1 does not carry more conceptual information (PCA remove/retain ablations, Fig. 3c-d).
- Local structures are highly aligned across architectures - ZCA-CKA of concept manifolds (mean-max ~0.86-0.87 on LAMA TREx vs 0.44-0.57 on general corpus) and high mutual k-NN overlap at corresponding depths.
- Local alignment emerges within ~5K training steps (~4% of Pythia training) whereas global geometry metrics stabilize only around ~25% of training.
Strengths
Controlled multi-architecture design
Comparisons use tokenizer-matched pairs at two scales (Pythia vs Mamba at 1.4B/2.8B; Falcon vs FalconMamba at 7B) plus hybrid Jamba2-3B, all analyzed at the residual-stream level (Section 2.1), which makes the cross-architecture contrast unusually clean for this literature.
Careful whitening methodology with artifact check
Section 5.1 computes ZCA whitening covariance globally from 10K matched tokens to avoid trivially isotropizing relation manifolds, and Appendix E's alpha-sweep shows the LAMA-vs-corpus alignment gap persists without whitening (layer mean 0.78 vs 0.52), directly ruling out the main confound.
Above-average stability reporting
Controls and triangulated metrics
Weaknesses
Probing lacks selectivity/control baselines
Training-dynamics claims rest on one model family
'Mean max' CKA statistic and thin per-relation data
Unexplained anomaly excluded from statistics
Attention-as-collapse-driver is observational
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.08692, cs.CL) asks whether transformers and selective state-space models induce different internal representation geometries. It compares five open models at the residual-stream level: Pythia-1.4B/2.8B and Falcon-7B (transformers), Mamba-1.4B/2.8B and FalconMamba-7B (SSMs), and hybrid Jamba2-3B. Four central claims are advanced: (1) **global divergence** - SSM residual streams are isotropic with high RankMe effective dimensionality, while transformers are anisotropic with a dominant PC1; (2) **functional equivalence** - matched effective capacity under rank-constrained autoencoders and matched concept-subspace dimensionality under probes; (3) **local convergence** - aligned concept manifolds (ZCA-CKA) and token neighborhoods (m-KNN); (4) **asymmetric training dynamics** - local alignment emerges by ~4% of Pythia training, global geometry stabilizes near ~25%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do Mamba-style SSMs and transformers learn the same internal representations?
According to this paper, they differ globally but align locally. Globally, SSM residual streams are isotropic with high effective dimensionality, while transformer representations are anisotropic and dominated by one principal component. Locally, concept manifolds (ZCA-CKA around 0.86-0.87 on LAMA TREx for trained model pairs) and token neighborhoods (mutual k-NN) match closely at corresponding network depths.
What is the evidence that SSMs are more isotropic than transformers?
Layer-wise RankMe, IsoScore, and PC1-explained-variance were computed over roughly 500K FineWeb-Edu tokens (Fig. 2): Mamba and FalconMamba maintain higher RankMe and IsoScore and far lower PC1 variance than size-matched Pythia and Falcon. In the hybrid Jamba2, attention layers spike PC1 dominance and subsequent Mamba layers reverse it, pointing to attention as the collapse driver.
Does the transformer's dominant principal direction store factual knowledge?
The paper argues no: removing or retaining the top-k PCA directions degrades LAMA TREx probe accuracy similarly in transformers and SSMs (Fig. 3c-d), and rank-constrained probes need similar subspace ranks in both architectures, suggesting PC1 reflects broad structural properties rather than extra conceptual content.
When during training do transformer and SSM representations start to align?
Using Pythia's public checkpoints, cross-architecture CKA and mutual k-NN reach near-final alignment by about 5,000 steps (~4% of training), while global geometry metrics such as IsoScore and PC1 variance keep changing until roughly 25% of training. The temporal claim rests on Pythia only, since SSM checkpoints are unavailable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric values embedded inside figure plots (e.g., exact probe accuracies in Fig. 3b-d, RankMe/IsoScore levels in Figs. 2, 6, 7, m-KNN levels in Fig. 5) are not recoverable from the extracted text; quantitative judgments for those rely on captions, the abstract, and appendix tables.
- The arXiv metadata abstract in the prompt is cut mid-sentence, but the complete abstract appears at the start of the paper text itself - no content is missing from the review basis.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.