Geopolitical Divisions Across Languages in Large Language Models
cs.AI · cs.CL · cs.CY
Quick verdict
LLM audit in 112 languages: Ukraine-war answers shift with language and track global politics; descriptive result solid, causal claims exceed evidence.
Summary
The paper audits three frontier models (GPT 5.6 Sol, Claude Sonnet 5, Gemini 3.8 Flash) on a bank of twenty statements about the Russia–Ukraine war, arranged in ten matched pro-Russian/pro-Ukrainian topic pairs, administered in 112 language and script conditions with ten repetitions, yielding 67,200 valid scores. Balance — mean agreement with pro-Russian minus pro-Ukrainian statements — leans toward Ukraine in every language but varies widely: Ukrainian is most pro-Ukraine (−70.49) and Sango least (−27.31). Averaged over countries via official-language mappings, the balances correlate with Pew Russia favourability (r = 0.569), UN non-support (r = 0.384) and Kiel bilateral aid to Ukraine relative to GDP (r = −0.707); the pattern recurs across models and after removing each statement pair. The author proposes information-warfare narratives entering training corpora as an explanation, without tracing training data. Main limitations: no preregistration; translations machine-checked without independent semantic validation; country observations non-independent because countries sharing official languages inherit identical scores; data and code not yet deposited; commercial models collected in September 2026.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Asking in different languages changes how the same frontier models assess matched statements about the Russia–Ukraine war (Ukrainian −70.49 vs Russian −44.97 balance points).
- The language-conditioned balance leans toward Ukraine in all 112 conditions, but its strength varies widely (−70.49 to −27.31) and maps onto worldwide political divisions when aggregated by countries' official languages.
- Country-level balances correlate with Pew Russia favourability (r = 0.569), UN non-support (r = 0.384) and Kiel aid to Ukraine relative to GDP (r = −0.707).
- The pattern recurs across GPT 5.6 Sol, Claude Sonnet 5 and Gemini 3.8 Flash and survives leave-one-pair-out removal of every statement pair.
- Information-warfare narratives entering multilingual training corpora are a plausible mechanism, though the study does not trace training data directly.
Strengths
Unusually broad language coverage
112 language/script conditions × 20 statements × 10 repetitions × 3 models = 67,200 valid scores (Methods: Models and API collection), moving beyond the two belligerents' languages that dominate prior work (refs 5–7) to a genuinely global sample.
Matched-pair design controls statement direction
Ten matched pro-Russian/pro-Ukrainian topic pairs (Table 1, Methods: Study design) let the balance metric cancel topic-specific agreement levels; the leave-one-pair-out check keeps country rank correlation ≥0.989, so no single topic drives the index.
Triangulation across three independent external benchmarks
Honest scoping of claims
Substantial sensitivity battery
Weaknesses
Language proficiency and translation quality are unaddressed confounds
Country observations are non-independent and the bootstrap cannot capture between-country variation
Causal interpretation of the information-warfare pathway is asserted, not tested
'Russia-leaning' is relative; the direction label invites misreading
No correction for multiplicity; inference ignores shared-mapping dependence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (arXiv 2609.20005, cs.AI/cs.CL/cs.CY) asks whether language alone — with no persona — changes the political slant of frontier-model answers. Three commercial models (GPT 5.6 Sol, Claude Sonnet 5, Gemini 3.8 Flash) rate twenty matched statements on the Russia–Ukraine war (ten pro-Russian/pro-Ukrainian topic pairs, Table 1) across 112 language/script conditions with ten repetitions: 67,200 valid scores. The central claims are (i) language conditions the balance between pro-Russian and pro-Ukrainian agreement, (ii) country-level balances, built via official-language mappings, track Pew Russia favourability (r = 0.569), UN ES-11 non-support (r = 0.384) and Kiel aid/GDP (r = −0.707), and (iii) information-warfare narratives in training corpora are a plausible route. Claim (iii) is explicitly untested; the paper concedes it "does not trace particular narratives through training data."
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do AI chatbots answer political questions differently depending on the language you ask in?
This study of GPT 5.6 Sol, Claude Sonnet 5 and Gemini 3.8 Flash finds they do. The same twenty statements about the Russia–Ukraine war were presented in 112 languages; agreement leaned toward the Ukrainian side in every language, but the strength of the lean varied widely — most strongly in Ukrainian (−70.49 points) and least in Sango (−27.31). The differences line up with each country's public opinion of Russia, UN votes, and aid to Ukraine.
Did the study find that the models give pro-Russian answers in some languages?
No. In all 112 language conditions the net balance leaned toward Ukraine. What varies is how strongly: languages spoken in countries closer to Russia (for example, Central African Republic, Mali, Algeria) produced relatively more Russia-leaning balances, while Ukrainian, Latvian, Lithuanian, Estonian and Polish produced the strongest pro-Ukraine balances.
Does this prove that information warfare or propaganda biased the models' training data?
No — and the author says so. The observed pattern is consistent with narratives spread through multilingual text entering training corpora, but the study did not trace any narratives through training data. Other mechanisms, such as differences in safety tuning or reinforcement learning across languages, were not tested and could also explain the pattern.
Can the results be reproduced, and is the data available?
The paper states that the data and code will be provided in a public repository, but as of this preprint version neither is deposited yet. The methods are documented in enough detail (exact model IDs, prompts, aggregation formulas) to re-run the audit once released, though commercial model behaviour may have changed since the September 2026 collection.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based on extracted text; Figures 1–4 are summarized from captions and surrounding text only, so visual properties (axis ranges, point clouds, outliers) could not be verified.
- Tables of per-language and per-country scores, the statement-bank translations, and the seven excluded language conditions are not in the extracted text and were not inspected.
- The referenced data and code repositories were stated as forthcoming and could not be checked.
- Model names/dates (GPT 5.6 Sol, Claude Sonnet 5, Gemini 3.8 Flash, September 2026) are taken as stated in the text; version drift may affect replication.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.