Why Pretraining Fails to Share Cross-Lingual Knowledge
Adam Gaber, Uriel Dolev, Elisabeth Fittschen, Bobby Cheng, Yuval Marton, Leshem Choshen
cs.CL · cs.AI · cs.LG
Quick verdict
Disjoint token spaces alone cause cross-lingual knowledge compartmentalization in pretraining; evidence is well controlled, but each result is one run.
Summary
The authors ask where LLMs' poor cross-lingual knowledge transfer originates. They pretrain bilingual models from scratch at 360M (SmolLM2) and 7B (Llama-3-derived) scale on English–Arabic and English–Russian corpora, inject a Fictional Knowledge Dataset of 2,048 fictive facts at exactly controlled per-language exposure counts, and introduce the CLeq score — the value of a foreign-language exposure relative to a native one, estimated by regression. Baseline bilingual pretraining yields CLeq of 0.9–5.9%, and neither code-switching-style language mixing (2.2%) nor activation alignment (2.5%) helps. In a clone setup (English1–English2) where identical language copies differ only by disjoint token IDs, transfer collapses to −2.6%, and partial embedding tying shows a sharp threshold (8.6% at 20%, 82.7% at 50%). Word-wise translation (WWT), mapping Arabic or Russian words into English token space with an invertible dictionary, raises CLeq to 12.6% (14× baseline) and 23.3% for Russian while improving English perplexity; controls show the mapping must be semantically faithful. Limitations: the Arabic corpus is machine-translated (translationese risk, partly mitigated by Russian replication), CLeq rests on a linear approximation, 2,048 facts give wide marginal confidence intervals, each configuration is a single pretraining run, and WWT's ~30% token-fertility overhead is unoptimized.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Knowledge compartmentalization originates during pretraining and persists under standard interventions (language mixing, activation alignment), with scale attenuating but not resolving it (0.9% → 5.9% CLeq from 360M to 7B).
- Disjoint token spaces alone suffice to induce compartmentalization, even between identical copies of the same language (CLeq −2.6% in the English1–English2 setup).
- A large amount of token sharing is required: generalization jumps sharply between 20% and 50% embedding tying, and coverage of shared tokens matters more than their vocabulary count.
- Word-wise translation into a shared token space improves cross-lingual knowledge generalization up to 14× (CLeq 12.6% En–Ar, 23.3% En–Ru) while also improving English perplexity.
- Token sharing helps only when the mapping is semantically faithful: shuffled dictionaries, pure transliteration, and permuted token identities all fail.
Strengths
Clone-language causal design
English1–English2 (§5.1) differs from standard pretraining in nothing but disjoint token IDs, and CLeq collapses to −2.6% (−0.1% even with identical corpora; −1.5% at 7B, §5.2) — the cleanest isolation of token space as the causal variable.
Controlled knowledge exposure with a purpose-built metric
FKD injection (§2.1, App. B) gives exact per-language exposure counts, validated by a log-like acquisition curve (App. B.5) and realized-vs-target robustness (Tab. D.4); CLeq (§2.2) normalizes by native learning, decoupling transfer from raw competence.
Semantic controls rule out the obvious alternative
Dose-response and replication breadth
Above-average statistical transparency
Weaknesses
One pretraining run per configuration
Translationese confound in the headline pair
Linear estimator on a log-like curve
Asymmetric transfer explained only post hoc
Overlooked nuance vs the 'fundamental barrier' framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks where cross-lingual knowledge compartmentalization in LLMs originates and answers: in pretraining, driven largely by disjoint token spaces. Three claims are advanced. (1) Compartmentalization emerges during standard bilingual pretraining and persists under the usual interventions — language mixing via code-switching reaches 2.2% CLeq and activation alignment 2.5%, against a 0.9% baseline (§4, App. E); scale attenuates but does not resolve it (5.9% at 7B, Tab. 2). (2) Disjoint token spaces are a root cause: in the English1–English2 clone setup, where identical language copies differ only in token IDs (§5.1), CLeq collapses to −2.6%, stays at −0.1% with identical corpora, and only reaches −1.5% at 7B (§5.2). (3) Word-wise translation (WWT) into a shared token space lifts CLeq to 12.6% for English–Arabic (14× baseline), 12.5% at 7B, and 23.3% for English–Russian, while improving English perplexity from 19.53 to 18.74 (§6.2, Tab. 2). The abstract calls disjoint tokens a 'fundamental barrier'; the clone-setting code-switching run (71.8% CLeq, Tab. H.1) shows the barrier is bridgeable when languages are otherwise identical, so 'fundamental for natural language pairs' would be the defensible phrasing.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do LLMs fail to transfer knowledge across languages?
This paper argues the main barrier is disjoint token vocabularies during pretraining. When two identical copies of English are trained with different token IDs, knowledge transfer is zero; when the languages share a token space through word-level translation mapping, transfer improves up to 14×.
What is the CLeq score in this paper?
CLeq (Cross-lingual equivalence) measures how valuable seeing a fact in one language is for recalling it in another, relative to seeing it natively — 100% means perfect transfer, 0% means complete compartmentalization. It is estimated by regressing recall accuracy on controlled exposure counts of injected fictive facts.
What is word-wise translation (WWT)?
WWT is a data-level mapping that replaces each Arabic (or Russian) word with its English-token counterpart using a large invertible dictionary (Buckwalter/GOST transliteration as fallback). Grammar and word order are untouched; the mapping is applied and inverted on the fly, so users read and write native-script text.
Does scaling up the model fix cross-lingual knowledge compartmentalization?
No. Moving from 360M to 7B parameters raises CLeq only from 0.9% to 5.9% in the natural bilingual setting, and in the controlled clone setup scaling gives no improvement at all (−2.6% to −1.5%). Most cross-lingual transfer value remains unrecovered.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as embedded text values from captions; visual details of Fig. 1/Fig. 2 could not be independently verified.
- The extraction omitted some spans (parts of §5.3–§6 transitions, App. I.1 opening, and middle reference pages); judgments rely on the retained text.
- Review covers arXiv v1 (16 Sep 2026), a non-peer-reviewed preprint.
- CJK renderings of the Arabic WWT examples in Tab. 1/Fig. I.1 appear garbled by extraction and were not used for assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.