Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3
Giang Son Nguyen, Nhi Ngoc-Yen Nguyen, Wray Buntine, Dung D. Le
cs.CL · cs.AI
Quick verdict
Reproduction study: SAE translation-feature recurrence overstates causal transfer; well-controlled Gemma 2/3 evidence but only 6 languages, 16k SAEs.
Summary
This reproduction-and-extension study revisits Wu et al.'s (2026) translation-initiation SAE features and asks whether they retain causal power when the prompt, source, and target languages change. Using the original three-stage pipeline (feature discovery at boundary tokens, PCA-based consistency filtering requiring 95% first-component variance, causal intervention at alpha=0/2), the authors run four discovery settings over English, Chinese, and Japanese on Gemma 2 2B IT with Gemma Scope SAEs, then repeat the pipeline on Gemma 3 4B IT with Gemma Scope 2 SAEs. In each model, 22 features recur across all four discovery settings, yet only one — (L10, 5717) and (L20, 2456) respectively — shows the expected switch signature across 23 evaluation settings built from WMT24++ (960 examples, 6 languages): amplification improves COMET (Gemma 2 gains +1.00 to +7.46; Gemma 3 up to +2.50, though non-significant in ar2en_ar) and ablation degrades it. The 21 remaining features form a matched null with small or inconsistent effects, and a control over 20 non-core features finds nothing above +0.65. Dose-response, task-specificity (XL-Sum, WikiANN), chrF++, and non-translation-rate checks support the conclusion. Main limitations: discovery restricted to three languages, 16k-width SAEs only, individual-feature interventions, and no multiple-comparison correction.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Multilingual feature recurrence is a weak proxy for causal transfer: in both Gemma 2 and Gemma 3, 21 of the 22 features recurring across four discovery settings show small, inconsistent, or opposite-sign intervention effects.
- One feature per model — Gemma 2's (L10, 5717) and Gemma 3's (L20, 2456) — functions as a language-agnostic translation-initiation switch, improving COMET under amplification and degrading under ablation across 23 language settings.
- The effect is task-specific rather than a generic multilingual boost: amplifying (L10, 5717) changes ROUGE-L on XL-Sum by only -0.011 to +0.014 and entity micro-F1 on WikiANN by -0.021 to +0.012.
- The discovery-and-intervention pipeline of Wu et al. (2026) transfers to Gemma 3 with an identical qualitative outcome despite differences in architecture, size, and training data.
Strengths
Matched-null causal design
The 21 features that recur across all four discovery settings but lack the switch signature (Tables 3 and 4) serve as a built-in control group that passed identical discovery and PCA filters, making the core claim that recurrence overstates causal transfer internally controlled rather than anecdotal.
Converging evidence beyond the headline metric
The single-feature conclusion is triangulated by chrF++ (Appendix E, Tables 10-11), fastText-based non-translation rates (Table 6: amplification cuts the rate by up to 13.7 points, ablation raises it by up to 17.3 points in Gemma 2), dose-response curves (Figure 1, peak at alpha=2-3), task-specificity probes on XL-Sum and WikiANN (Tables 8-9), and the non-core feature control (Table 7, max +0.65 vs +4.61 for (L10, 5717)).
Cross-model replication with an identical qualitative outcome
Significance disclosure and reproducibility hygiene
Weaknesses
Abstract overstates Gemma 3 consistency
Narrow linguistic footprint behind a language-agnostic claim
No multiple-comparison control
Single SAE-width confound
Combined interventions untested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a reproduction-and-extension study of Wu et al. (2026, AAAI), who discovered translation-initiation SAE features in Gemma 2 under English-centric conditions. The authors extend the original single discovery setting (en2zh_en) to four (en2zh_en, zh2en_zh, zh2ja_zh, zh2ja_en), expand evaluation from 4 to 23 src2tgt_prompt settings over six languages (en, zh, ja, ar, ru, vi), and add a second model (Gemma 3 4B IT with Gemma Scope 2 SAEs alongside Gemma 2 2B IT with Gemma Scope). Three claims anchor the paper: (1) feature recurrence across multilingual discovery settings is a weak proxy for causal transfer (21 of 22 recurring features fail per model); (2) one feature per model — (L10, 5717) and (L20, 2456) — acts as a language-agnostic translation-initiation switch; (3) the pipeline transfers to Gemma 3 with an identical outcome.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That sparse-autoencoder features which repeatedly appear across multilingual discovery settings are usually not causally important: in Gemma 2 and Gemma 3, 22 features recurred across four language settings, but only one per model actually changed translation behavior when amplified or ablated. Feature recurrence alone therefore overstates cross-lingual causal transfer.
Which specific SAE features are identified as causal?
Feature (L10, 5717) in Gemma 2 2B IT and feature (L20, 2456) in Gemma 3 4B IT, both from layer-10/layer-20 residual-stream SAEs. Amplifying them improved COMET translation scores in nearly all of 23 language settings and ablating them degraded scores, while the other 21 recurring features had small or inconsistent effects.
Is the code and data available?
The paper states that code is available at github.com/giangson19/recurrence-is-not-enough. All models (Gemma 2 2B IT, Gemma 3 4B IT), SAEs (Gemma Scope and Gemma Scope 2, 16k width), and datasets (WMT24++, XL-Sum, WikiANN) are public, and experiments ran on two A100 40GB GPUs.
How large are the reported effects?
For Gemma 2's (L10, 5717), amplification raised COMET by +1.00 to +7.46 points across the 23 settings and ablation lowered it in every setting (up to -8.93). For Gemma 3's (L20, 2456), amplification gains reached +2.50 but were smaller overall (attributed to higher baselines) and non-significant in ar2en_ar; ablation degraded 22 of 23 settings.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.