Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation

Summary

The paper asks what happens to word-order information in late transformer layers, using Modern Greek object-relative minimal pairs in which the embedded clause appears in canonical SVO or scrambled VSO order with identical propositional meaning. Three Greek-tuned 8B models were tested (Llama-Krikri-8B-Base primary; Instruct and Plutus variants in appendices) on 128 template-generated sentences plus a 1024-sentence sensitivity set. From each layer's post-clause hidden states, four distributional statistics (mean, variance, skewness, kurtosis over tokens) feed an L2 logistic probe; performance is ROC-AUC under stratified 10-fold cross-validation with cluster-based permutation tests (1000 permutations, α=0.01). Post-clause decoding peaks in middle layers; a GAT-style probe trained on pooled late layers (20–31) transfers above chance across a middle-layer band (roughly 5–19) but falls significantly below chance on early layers, classifying 99.3% of non-canonical sentences as canonical (49.3% overall accuracy). Probe-coefficient sign scores flip around layer 22, which the authors interpret as directional recoding toward canonical order rather than information loss. Limitations: SVO and canonicality are co-extensive; probes are linear over coarse four-scalar summaries; no causal intervention is performed; one language and construction. Code, stimuli, and activations are publicly available on OSF.