SharedSAE: One Feature Dictionary Across Language Models
Daniil Ognev, Célian Vasson, Lijie Hu, Kentaro Inui, Benjamin Heinzerling
cs.LG · cs.CL
Quick verdict
A shared SAE dictionary across four 1B LLMs keeps 96.6% of dedicated-SAE variance with transferable labels; well-controlled and convincing at this small scale.
Summary
SharedSAE is a sparse-autoencoder architecture in which a single feature dictionary serves several language models at once: every model keeps its own encoder-decoder pair, tokenizer alignment supplies matching activation windows, and a joint TopK over normalized votes lets models agree on which latents fire while keeping their own magnitudes. Vote-source dropout trains each model to select latents alone, making single-model inference possible where prior shared-dictionary methods (USAE, SPARC) need all participants. On four 1B-scale base models with distinct tokenizers (Qwen3, Llama-3.2, OLMo-2, Gemma-3), the shared dictionary retains 96.6% of dedicated SAEs' mean explained variance; spliced behavior lands within 0.006 mean CE-recovered of the converged dedicated splice; and per-latent cross-model correlations reach a median of 0.79 under solo selection versus 0.43-0.45 for post-hoc matched separate SAEs. Descriptions validated in one model transfer substantially to the others (FADE clarity 0.85-0.89), though functional (causal) alignment stays partial. With the dictionary frozen, new models — including hybrid-attention Qwen3.5 and Tencent's Hunyuan-1.8B — adapt in under an hour to 0.972-1.010x their dedicated-SAE quality. Limitations: only four ~1B English-only models, one layer (~85% depth), a 32,768-latent dictionary, and sharedness metrics partly optimized by the training objective itself.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single SharedSAE dictionary with per-model encoder-decoder pairs retains 96.6% of dedicated per-model SAEs' mean explained variance across four 1B-scale models with distinct tokenizers.
- Under solo TopK selection, SharedSAE's per-latent cross-model activation correlations reach a median of 0.79, about 1.8x the 0.43-0.45 achieved by post-hoc matching of separate SAEs.
- Latent descriptions validated in one model transfer substantially to the others (cross-stream FADE clarity 0.85-0.89), while functional (causal) alignment remains only partial.
- With the dictionary frozen, new models — held-out, post-trained, and cross-architecture — adapt in 45-67 minutes to 0.972-1.010x their own dedicated SAE reconstruction quality while keeping existing latents and descriptions fixed.
- Vote-source dropout is the component that enables single-model inference: without it, the adapted hybrid-attention Qwen3.5 reaches only 0.436 of its dedicated solo quality (Table 4).
Strengths
Criteria fixed in advance, every method scored on them
Figure 1's criteria table enumerates the shared-SAE requirements (FVE from the shared space, spliced KL, per-token co-selection, top-50 overlap, cross-model correlation, description fixation under adaptation, solo inference) and fills it in for post-hoc matching, USAE, SPARC, and SharedSAE — including cells where competitors win (post-hoc splice KL 0.27 vs SharedSAE's 0.33).
Control battery well beyond field norm
100-draw frequency-matched permutation nulls; ridge-map floor of 0.610 vs 0.726 shared-code cross-FVE at the 8-epoch protocol (Appendix D.4); the 0.77 within-model seed-pair post-hoc ceiling (Figure 11); a planted-ground-truth synthetic check at universality ratio 1.000 (Appendix J); a Wikipedia out-of-corpus re-evaluation (median correlation 0.72, Appendix N); budget-matched and mid-depth arms (Tables 7, 13); three seed replicates within 0.003 FVE.
Negative and null results reported rather than buried
Unusually complete reproducibility scaffolding
Practical payoff demonstrated on genuinely new models
Weaknesses
Scope is narrow relative to the motivating problem
Metric circularity on two of three sharedness criteria
Functional alignment is much weaker than semantic alignment
Some headline cross-reconstruction numbers use an evaluation-time oracle
On behavior and semantics the result is parity, not superiority
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SharedSAE (arXiv 2609.04344) proposes replacing per-model sparse autoencoders with one shared dictionary of L=32,768 latents plus per-model encoder-decoder pairs, trained jointly over four 1B-scale base models (Qwen3-1.7B, Llama-3.2-1B, OLMo-2-1B, Gemma-3-1B) with four distinct tokenizers, hooked at ~85% depth on FineWeb-Edu (4M aligned positions, shard 9 held out). The claims are explicitly scoped: a shared dictionary that (i) reconstructs each model near dedicated-SAE quality, (ii) is genuinely shared (same latents chosen, correlated magnitudes, same meaning), (iii) supports single-model inference, and (iv) accepts new models via frozen-dictionary adaptation while keeping descriptions fixed. Section 4 opens by conceding that two of the three sharedness criteria are optimized by training — an unusually candid framing the evaluation then has to work around.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SharedSAE in one sentence?
SharedSAE is a sparse-autoencoder design in which one shared dictionary of interpretable features (latents) serves several language models at once, with each model connecting to it through its own small encoder-decoder pair.
How much quality is lost compared to training a separate SAE per model?
On the paper's four 1B-scale models, the shared dictionary retains 96.6% of dedicated SAEs' mean explained variance, and spliced-in behavior stays within 0.006 mean CE-recovered of the converged dedicated splice, beating budget-matched (8-epoch) dedicated SAEs on three of four streams.
Can SharedSAE run with a single model, and can a new model be added later?
Yes to both: vote-source dropout during training makes single-model selection in-distribution, and after the dictionary is frozen a new model trains only its encoder, decoder, and biases (45-49 minutes per adaptation in the paper) while reusing all existing latent descriptions.
What are the main limitations of the SharedSAE paper?
Evidence covers four English-only ~1B base models at one layer (~85% depth) with a 32,768-latent dictionary; cross-model functional (causal) alignment is only partial (faithfulness correlations 0.37-0.46); and two of the three sharedness metrics are partly optimized by the training objective itself, which the authors acknowledge.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt contains several [...sampling gap...] skips: the rendered contents of Table 1, Table 2, Table 6, and parts of Section 4.4 and Appendices C-D were not visible; numbers attributed to them were taken from in-text restatements, which were internally consistent.
- No statement releasing the authors' own SharedSAE code or aligned activation stores appears in the provided text (baseline provenance is described in Appendix B); a release link may exist in a skipped section, so codeAvailable=false reflects the excerpt, not a verified absence.
- PDF extraction renders some figure legends as run-on text (e.g., 'ASAE' standing in for SharedSAE in Figures 3-6); plot-level values were used only where corroborated by prose or other tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.