A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failures
Oskar Holmström, Marcel Bollmann, Marco Kuhlmann
cs.CL
Quick verdict
Solid comparison of 4 sharing metrics on 21 models: anisotropy explains disagreement, only ILO predicts transfer; n=5 families limits generalization.
Summary
The paper systematically compares four cross-lingual sharing metrics — CKA, ANC, GMM dominance per token (λ), and ILO — across 21 decoder-only base models from five families (BLOOM, OPT, XGLM, Qwen3, SmolLM3; 125M–14B parameters), correlating each with zero-shot cross-lingual transfer averaged over five tasks (Belebele, XNLI, XCSR, SIB-200, XQuAD) after English-only finetuning. The metrics disagree sharply, and the authors trace the disagreement to anisotropy: the variance-ranked PCA used by λ retains only two components in high-λ models, while language probes show language identity relocates to the discarded residual subspace (96–99% accuracy where the principal subspace falls to chance). Three sharing patterns emerge (concentrated; distributed, as in XGLM; minimal, as in OPT). Only ILO's correlation with transfer (Spearman ρ=0.90; partial ρ=0.84 controlling size) survives Benjamini–Hochberg per-task correction and a within-family permutation test (p=.014); ANC (ρ=0.84) rests on between-family contrasts and CKA shows no association. The authors recommend ILO as the primary metric, reported alongside anisotropy diagnostics. Stated limitations: an effective between-family sample of five, decoder-only scope, ten high-resource languages, an observational design, and no systematic hyperparameter sensitivity analysis.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Disagreement among the four sharing metrics traces to anisotropy: variance-ranked PCA collapses the measurement space to npca=2 in high-λ models while language identity lives in the discarded residual (96–99% probe accuracy).
- GMM dominance per token (λ) conflates three distinct sharing patterns — concentrated, distributed (XGLM), and minimal (OPT) — because its pipeline requires a low-dimensional clustering plane.
- Only ILO's association with zero-shot cross-lingual transfer (Spearman ρ=0.90; partial ρ=0.84) survives BH per-task correction, size control, and a within-family permutation test (p=.014); ANC's association rests on between-family contrasts and CKA shows none at any level.
- Sharing can emerge post-hoc: instruction tuning induces the collapsed-dimensionality regime in BLOOMZ-560m (λ 0.21→0.84), and during BLOOM pretraining sharing appears via abrupt phase transitions (λ 0.13→0.79 between steps 10K–100K in BLOOM-1b1).
- Geometric sharing scores should always be reported alongside anisotropy diagnostics such as effective dimensionality and mean random cosine similarity.
Strengths
Controlled like-for-like comparison design
λ and ILO are computed on the identical balanced 10,000-token-per-language FLORES sample, removing tokenizer-fertility bias; the input-unit confound is tested directly — CKA and ANC share sentence embeddings yet split on transfer (ρ=0.14 vs 0.84), and λ and ILO share token inputs yet invert on XGLM (λ=0.03 vs ILO=0.44, Appendix B.1).
Transfer-independent mechanistic diagnosis
Language probes (§4.2, Fig. 3) show identity is relocated, not destroyed: principal-subspace accuracy drops to ~chance (0.09) while the residual subspace reaches 96–99% at high-λ layers across BLOOM, Qwen3 and SmolLM3; the ILOc centering control (mean |Δ|=0.003, max 0.010, Table 8) empirically rules out anisotropy inflating ILO in these models.
Statistical hygiene unusual at n=21
Candid, specific Limitations section
Thorough reproducibility inputs
Weaknesses
Criterion proximity threatens the ILO ranking
Headline claim hangs on a single borderline test in a five-family design
λ is a modified reimplementation of Shani & Basirat (2025)
Tokenizer-overlap confound only partially handled
No code or data availability statement visible
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether cross-lingual sharing metrics measure the same thing, and which, if any, can be trusted. It compares four representative metrics — linear CKA (global geometry), ANC (neuron-level correlation), GMM dominance per token λ (distributional, Shani & Basirat 2025), and ILO (neighborhood overlap, Wilie et al. 2025) — on 21 decoder-only base models from five families (BLOOM, OPT, XGLM, Qwen3, SmolLM3; 125M–14B), plus 11 instruction-tuned variants and 59 checkpoints for characterization. Functional validation is zero-shot cross-lingual transfer after English-only finetuning, averaged over five tasks (Belebele, XNLI, XCSR, SIB-200, XQuAD). Three claims anchor the paper: (1) metric disagreement is systematic and traces to anisotropy; (2) λ's 98%-variance PCA pipeline collapses the measurement space, and language identity is relocated to the discarded residual (probe accuracy 96–99% there while the principal subspace falls to ~chance, §4.2); (3) only ILO predicts transfer robustly (ρ=0.90), so ILO should be the primary metric, always reported with anisotropy diagnostics.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which metric should I use to measure cross-lingual sharing in LLM representations?
According to this paper, ILO (Interlingual Local Overlap), a k-nearest-neighbor overlap metric computed on the full hidden state. It was the only one of four tested metrics (CKA, ANC, GMM dominance λ, ILO) whose correlation with zero-shot cross-lingual transfer (Spearman ρ=0.90 across 21 decoder models) survived controls for model size, model family, and per-task variation. The authors advise reporting ILO together with anisotropy diagnostics such as effective PCA dimensionality.
What is anisotropy and why does it break cross-lingual sharing metrics?
Anisotropy is the tendency of contextual embeddings to occupy a narrow cone in representation space, so random vectors already have high cosine similarity (0.86–0.99 in the studied models). It inflates similarity-based scores and concentrates variance in very few PCA directions: in high-sharing models only 2 principal components held 98% of variance, while probes showed language identity actually lived in the thousands of discarded directions — so metrics that project onto dominant variance directions end up measuring a subspace from which language has been removed.
Why did GMM dominance (λ) fail on XGLM in this study?
For XGLM, 98% of variance is spread over thousands of PCA components (e.g., 1,702 at XGLM-4.5B's peak layer), so no low-dimensional clustering plane exists for the GMM to fit, and λ stays near zero (0.03) by construction — even though XGLM-4.5B's ILO is 0.44 and its transfer score (0.55) approaches BLOOM-7.1B (0.60). Probes show language information remained in the residual subspace that λ's projection discards.
Does ILO work for low-resource languages?
Unknown from this paper. All ten evaluation languages are high-resource (three are Romance), chosen because all five benchmarks and all five model families cover them. The authors explicitly state they cannot say whether ILO remains the most reliable metric, or whether λ fails the same way, for languages thinly represented in pretraining, and leave this to future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with four marked [...sampling gap...] regions (parts of §3.5, the references, Appendix A.5–A.6, parts of the B.6–B.7 tables, and C.2–C.3); every judgment in this review rests on content actually visible in the excerpt.
- No code or data availability statement appears in the sampled text; whether one exists in a skipped region could not be verified, hence codeAvailable=false reflects the excerpt rather than a verified absence in the full paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.