We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation
Svetlana Gorovaia, Angelica Henestrosa, Ivan P. Yamshchikov
cs.CL · cs.AI
Quick verdict
Single charged words steer LLM translation across 15,680 prompts; novel cross-family design, but purely descriptive analysis lacks significance tests.
Summary
The paper asks whether a single politically charged word can trigger implicit political alignment in an ostensibly apolitical task. Eight models — GPT-5, Claude Sonnet 4.5, Gemini 3.5 Flash Lite (US), Mistral Large (EU), GLM-5, Qwen 3.7 Plus, Kimi K3, DeepSeek v4 Flash (China) — were asked to translate sixteen national dishes plus a neutral pancake control into 'the language of the aggressor / enemy / coloniser / neighbour', with the target language unnamed, across 16 instruction languages plus English and 10 repeats at temperature 0: 15,680 responses annotated by GPT-4o into a six-way taxonomy. Models largely resolve rather than refuse: Russian is the dominant sink (in-degree up to nine source languages), and post-Soviet→Russian, Hebrew–Arabic, Portuguese–Spanish, and Thai–Lao edges are universal. Chinese models comply and cluster tightly (mean switching distance ~0.12–0.13); Claude (11 graph edges) and GPT-5 (25) hedge with vague justifications; Mistral Large is a distinct sparse profile that nearly falls silent under the coloniser frame; conditional on compliance, GPT-5 resembles the Chinese cluster. The coloniser frame maximally separates models. Limitations: about ten prompts were LLM-translated without native verification; annotation relies on a single LLM judge; the analysis is purely descriptive with no inferential statistics; and the stated 196 combinations do not match the reported 4×17×2=136 design. Russian dominance may also reflect training-data frequency.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 6 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single politically charged framing term is sufficient to modulate LLM behavior in an otherwise apolitical recipe-translation task across all eight models tested.
- Models overwhelmingly resolve the ambiguous target language rather than declining or asking for clarification, with Russian as the dominant inferred destination (in-degree up to nine source languages).
- Response profiles cluster by model family: Chinese models comply silently and tightly cluster (mean pairwise switching distance ~0.12–0.13), Western models (Claude, GPT-5) hedge with vague justifications, and Mistral Large forms a distinct sparse profile that nearly drops the coloniser frame.
- Conditional on compliance, GPT-5's language-switching distribution is closer to the Chinese cluster than to the American one, and the coloniser framing maximally separates models while the neutral neighbour framing minimally does.
- LLM translations can embed implicit political judgments without any explicit signal to the end user in conflict-adjacent contexts.
Strengths
Scale with a fully crossed design
15,680 responses from 8 model families across 4 framing terms × 17 instruction languages × 2 recipe variants × 10 repeats at temperature 0 (§3.2) — unusual breadth for a bias audit, spanning US, EU, and Chinese labs simultaneously.
Probe isolates implicit inference
Leaving the target language unnamed forces the model to resolve 'the aggressor's language' itself (§3.1, Figure 1), and the neutral pancake variant separates dish-driven association from framing-term effects — supported in §4.2 where dominant-target shares rise for the pancake.
One crisp, checkable headline result
Quantified family clustering
Candid limitations section
Weaknesses
No inferential statistics anywhere
Arithmetic inconsistency in the design
Unvalidated LLM-as-judge annotation
Uneven stimulus fidelity
Interpretive tension around the Russian sink
Editorial defects suggest thin proofreading
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper probes whether a single politically charged word can steer LLM behavior in an apolitical task. Eight production models — GPT-5, Claude Sonnet 4.5, Gemini 3.5 Flash Lite (US); Mistral Large (EU); GLM-5, Qwen 3.7 Plus, Kimi K3, DeepSeek v4 Flash (China) — are asked to translate a recipe into "the language of the {aggressor, enemy, coloniser, neighbour}", with the target language deliberately unnamed (§3.1, Figure 1). The design crosses four framing terms × 16 instruction languages (+English baseline) × two recipe variants (16 national dishes per Table 1 plus a neutral pancake control) × 10 repeats at temperature 0, yielding a claimed 15,680 responses (§3.2). Central claims (Abstract, §4–5): models resolve rather than refuse; Russian is the dominant translation sink; behavior clusters by model family; framing sensitivity is universal; Mistral Large is an outlier profile. The authors frame the work as "fully exploratory" with no directional predictions (Limitations).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 6 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the 'We're Cooked!' paper find about LLM political bias?
When asked to translate a recipe into 'the language of the aggressor / enemy / coloniser / neighbour' without naming the language, most models resolve the ambiguity rather than refuse. Russian was the most common inferred destination (receiving translations from up to nine source languages), Chinese-family models complied and behaved very similarly to each other, GPT-5 and Claude hedged with vague justifications, and Mistral Large showed a unique sparse profile that nearly stopped translating under the 'coloniser' framing.
Which models were tested and how many responses were collected?
Eight models: GPT-5, Claude Sonnet 4.5, and Gemini 3.5 Flash Lite (US); Mistral Large (EU); GLM-5, Qwen 3.7 Plus, Kimi K3, and DeepSeek v4 Flash (China). Across 16 instruction languages plus English, four framing terms, and two recipe variants (national dish and neutral pancake) with 10 repeats at temperature 0, the study reports 15,680 responses, labeled by GPT-4o.
Does the Russian-language result prove political bias?
No — and the authors say so themselves. A dominant target language can reflect training-data frequency rather than a deliberate stance, about ten of the sixteen non-English prompts were machine-translated and never verified by native speakers, and the study is explicitly exploratory with no inferential statistics, so the patterns are descriptive rather than statistically tested.
Is the code or dataset from this paper available?
The paper provides only an interactive web page visualizing the translation graphs; no code release, raw-response dataset, full prompt strings, or complete recipe texts are included in the text, which limits independent replication.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Cell-level values inside Figures 10, 12, and 13 were extracted as garbled number streams without reliable row/column alignment; only figure-adjacent prose (e.g., edge counts, in-degree nine, JSD means 0.12/0.13) was cited.
- The provided text ends at the Appendix Figure 13 caption; any paper content beyond that point was not available in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.