Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs
Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov, Josef van Genabith, Simon Ostermann
cs.CL
Quick verdict
Claims mLLM translation fixes target syntax before language before meaning via word-order heads; careful design, but small effects and no statistical testing.
Summary
The paper asks whether commitment to grammatical structure is a distinct stage in how multilingual LLMs translate. Extending Dumas et al.'s (2025) concept/language split, the authors add word order as a third factor and test three models (mGPT 1.3B, Aya Expanse 8B, Llama 3 8B) on three synthetic minimal-pair datasets (197 noun-phrase, 154 SVO, 31 modal-verb sentences across 8, 5, and 4 languages). LogitLens probing shows intermediate predictions follow target-language word order while still favoring English tokens. Layer-wise activation patching between base/plant prompts varying syntax, language, and concept tracks probability over 2^3 factor combinations; maximum probability shifts occur in monotonic syntax→language→concept order in eight of nine model-dataset combinations (exception: Llama 3 on SVO, where syntax and language switch together at layer 14). Within the syntax-switch layer, single attention heads (Llama 3: 14.25; Aya: 15.0) preferentially shift word order, and mean-activation patching raises KL divergence only when patch languages differ in word order, not language identity; activations nevertheless do not cluster geometrically by word order. Limitations: no statistical uncertainty is reported, mGPT's syntax-only effects are near zero and its top head conflates syntax with language, data are template-synthetic via DeepL, and a 27-sentence FLORES check is inconclusive.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Translation in mLLMs proceeds as a staged S→L→C process: target-side word order is committed before surface language, which precedes lexical content, in eight of nine model-dataset combinations (Table 3).
- Intermediate representations already follow target-language word order while still predicting English tokens, so English acts as a conceptual rather than syntactic anchor (Sec. 4.1, Fig. 3).
- Syntactic commitment localizes to specific attention heads (Llama 3: 14.25; Aya Expanse: 15.0; mGPT: distributed across 11.2/11.7) that shift word order while remaining largely invariant to language identity (Sec. 4.3, Eq. 1).
- S-heads encode syntax functionally, not geometrically: mean-activation patching shifts word order only across different-order languages (Fig. 7), yet mean activations do not cluster by word order (Fig. 8).
Strengths
Clean three-way factorial design
Base–plant patching tracking all 2^3 S/L/C combinations (Table 2, Sec. 3.4) gives an internally valid dissociation logic and extends Dumas et al. (2025)'s two-way concept/language split with a controlled third factor.
Converging multi-method evidence
LogitLens (Fig. 3, App. D), layer patching (Table 3), head-level ratio patching (Eq. 1, Fig. 4), same-vs-different word-order mean patching with KL controls (Sec. 5, Fig. 7), and a geometric null check (Fig. 8) all point in the same direction.
Typologically broad controlled data
Exemplary exception reporting
Weaknesses
No statistical uncertainty anywhere
Core dissociation fails in the only balanced-multilingual model
S is fully determined by L by construction
Template-synthetic data with sensitive prompting
Head localization is layer-conditional
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper extends the two-way decomposition of translation in multilingual LLMs (conceptual content vs. output language; Dumas et al., 2025) into a three-way one by adding syntactic structure (word order) as an independently trackable factor. Three central claims: (1) translation proceeds as a staged S→L→C process — target-side word order is committed before surface language, which precedes lexical content; (2) intermediate representations follow target-language word order while still predicting English tokens, so English acts as a conceptual rather than syntactic anchor (Sec. 4.1, Fig. 3); (3) syntactic commitment localizes to attention heads (Llama 3: head 14.25; Aya Expanse: 15.0; mGPT: distributed across 11.2/11.7) that are word-order-sensitive but largely language-invariant, and functional rather than geometric (no clustering by word order in Fig. 8). The scope is deliberately narrow: noun-phrase ordering, SVO/SOV, and modal-verb placement in three models.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do multilingual LLMs decide the target language or the sentence structure first when translating?
This paper reports the latter: in the tested models (Llama 3 8B, Aya Expanse 8B, mGPT 1.3B), intermediate layers commit to the target language's word order (syntax) before committing to which language the output tokens will be written in, and lexical meaning is resolved last — a syntax→language→concept sequence observed in 8 of 9 model–dataset combinations via activation patching.
What are 'S-heads' or word-order attention heads in LLMs?
The authors identify attention heads — head 14.25 in Llama 3 8B and head 15.0 in Aya Expanse — whose patching shifts the predicted word order (e.g., adjective-first vs. noun-first) while barely changing the output language. Patching with mean activations from languages sharing the target's word order leaves predictions nearly unchanged, suggesting these heads encode syntax rather than language identity.
Do LLMs think in English when translating?
Consistent with Wendler et al. (2024), this paper's LogitLens analysis shows intermediate predictions favor English tokens — but crucially those English tokens already follow the target language's word order (e.g., English 'apple' predicted where French requires noun-first 'pomme rouge'). The authors conclude English serves as a conceptual anchor, not a syntactic one.
Is this evidence based on natural text?
Mostly no: the main experiments use template-generated minimal sentences (197 noun-phrase, 154 SVO, 31 modal-verb items) translated via DeepL and manually checked. A small check with 27 FLORES-200 sentences was inconclusive due to dataset size, which the authors acknowledge as a limitation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal numeric values (e.g., Fig. 7 KL-divergence heatmaps, Fig. 8 cosine-similarity matrix, Fig. 4 probability-ratio curves) are not recoverable from text extraction; assessment of those panels relies on the authors' textual descriptions and stated significance of patterns.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.