Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs
Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut
cs.CL · cs.AI · cs.LG
Quick verdict
Careful probing study: GMM and decoding-based latent-language probes disagree, exposing different facets of multilingual computation, not one lingua franca.
Summary
This paper reframes latent language identification (LLID) in multilingual LLMs as a measurement problem: instead of asking whether models 'think in English,' the authors ask whether different probes measure the same phenomenon. On identical hidden states they compare a per-layer GMM representation probe with decoding-based probes (raw logit lens and tuned lens, with argmax or top-p rollouts classified by GlotLID), across up to 12 decoder-only models from GPT-2 to the 7–9B class (Llama-2/3.1, Aya-23, EuroLLM, Mistral-Nemo, OLMo-2, Apertus), controlled copy/cloze/translation tasks, open-ended PUD and INCLUDE prompts, three QA domains, and token-aligned OLMo-2/Apertus checkpoints spanning ~210B to ~4T training tokens, in up to 27 languages. Probes systematically disagree: the GMM probe shows earlier cross-lingual mixing and weaker English bias, while decoding-based estimates stay sharper and more English-biased. These differences track multilinguality and training progression, are comparatively stable across domains and language inventories, and instruction tuning shifts little (contrary to the authors' expectation). Limitations: no ground truth for 'latent language,' the representation family is instantiated only via GMM, checkpoint analysis covers two model families, and generalization to other scales or architectures is untested. Code is released; the paper convincingly argues probe choice is part of the claim.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Representation-based (GMM) and decoding-based LLID probes applied to the same hidden states systematically disagree: GMM shows earlier cross-lingual mixing and weaker English bias; decoding retains sharper, more English-biased signals.
- Probe disagreement tracks model multilinguality and pretraining progression (OLMo-2/Apertus, ~210B–4T tokens) but is comparatively stable across domains and language inventories.
- Instruction tuning produces little systematic shift toward English in LLID estimates, and its effects are smaller than checkpoint-driven changes.
- Current LLID probes expose complementary aspects of multilingual computation rather than revealing a single internal lingua franca — the choice of probe is part of the claim.
Strengths
Measurement-problem framing
§2.1 explicitly defines q as a diagnostic estimate of an unobserved variable Z, not ground truth — the right epistemic stance for a field where 'the model thinks in English' claims (Wendler et al., 2024) rest on single probes; §8 lands the takeaway that probe choice is part of the claim.
Breadth and staging of the comparison
Up to 12 models (Table 2), three controlled tasks plus two open-ended corpora (PUD variants, INCLUDE domains), up to 27 languages, and token-aligned checkpoints from ~210B to ~4T tokens (Table 3); §4→§5→§6 builds from the favorable controlled case to generalization before the head-to-head probe comparison.
Calibration hygiene
Honest reporting of a surprising null result
Weaknesses
Representation family is a single estimator with unexamined confounds
Statistical reporting is descriptive only
Mild circularity in the multilinguality proxy
Narrow scale and architecture
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (arXiv 2609.00155, EPFL) attacks a load-bearing assumption in multilingual-LLM interpretability: that latent language identification (LLID) probes — the instruments behind claims like "Llamas work in English" (Wendler et al., 2024) — measure a single phenomenon. §2.1 formalizes LLID as estimating q(ℓ) ≈ Pr(Z=ℓ|h) where Z is unobserved, explicitly demoting q to "a diagnostic estimate, not ground-truth evidence." On identical hidden states the authors compare a representation-based probe (per-layer GMM, one component per language, after Shani & Basirat 2025) against decoding-based probes (raw logit lens; tuned lens of Belrose et al., with argmax/top-p rollouts classified by GlotLID). Claims: (1) probes systematically disagree — GMM shows earlier cross-lingual mixing and weaker English bias, decoding stays sharper and more English-biased; (2) disagreement tracks multilinguality and training progression; (3) domains and language inventories matter less (Fig. 16); (4) instruction tuning barely matters (§6, Figs. 20–21); (5) therefore "the choice of probe is itself part of the claim" (§8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do multilingual LLMs actually think in English?
This paper argues the question has no probe-independent answer. On the same hidden states, GMM-based probes of activation geometry show earlier cross-language mixing and weaker English bias, while decoding-based probes (logit lens/tuned lens) stay sharper and more English-biased. The authors conclude current probes measure different aspects of multilingual processing rather than revealing one internal lingua franca.
What is the difference between logit lens, tuned lens, and GMM probing?
Logit lens projects an intermediate hidden state directly through the model's output head to get a vocabulary distribution; tuned lens first applies a learned per-layer affine mapping before the output head; a GMM probe instead fits per-language Gaussian components to hidden states and reports which language cluster a state resembles. The paper shows these can give different 'latent language' answers for the identical state.
Which models, languages, and data were used?
Up to 12 decoder-only models from GPT-2 to the 7–9B class (Llama-2/3.1, Aya-23, EuroLLM, Mistral-Nemo, OLMo-2, Apertus and instruct variants), up to 27 languages, on controlled copy/cloze/translation tasks, PUD parallel sentences (9/21-language variants plus a 6-language UD extension), and INCLUDE QA across three domains, plus OLMo-2/Apertus checkpoints spanning ~210B–4T training tokens.
Does instruction tuning make models more English-centric internally?
Not in this study: base and instruction-tuned variants showed only modest differences in latent-language estimates (Fig. 4; Figs. 20–21), contrary to the authors' initial expectation, and pretraining-checkpoint effects were considerably larger than any base-vs-instruct shift.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure interiors are not recoverable from text extraction (only captions, axis labels, and the prose descriptions of curves); judgments about specific curve shapes rely on the paper's narrative descriptions of Figs. 3–6 and 12–21.
- The released code repository (github.com/bayazitdeniz/latent-lid) is cited in footnote 1 but its contents could not be verified in this offline review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.