Script Choice in LLMs: Evidence for Late-Layer Commitment
David Kletz, Sandra Mitrović, Itay Sabato, Ljiljana Dolamić, Fabio Rinaldi
cs.CL
Quick verdict
Shows early script encoding but late output-script commitment in Qwen; solid probe+logit-lens evidence, yet the depth-bottleneck claim remains correlational.
Summary
The paper asks where in an LLM's layers script knowledge — input-script recognition, instructed output-script, and the script actually generated — becomes accessible. Using the Qwen 2.5 Instruct family (0.5B–32B; 24–64 layers), the authors run logistic-regression probes on last-prompt-token hidden states and a logit-lens analysis over 8,304 prompts spanning nine scripts and 17 language-to-script mappings. Probes show input script decodable from layer 1 (~0.98 balanced accuracy) and requested output script by layers 3–5 (1.00) in all sizes, whereas the used-output script only becomes readable late (peaking at 0.77–0.86, at 82–100% of depth). The logit lens shows the top-1 token commits to Latin within 1–2 layers but to non-Latin targets only above ~85% depth (e.g., Cyrillic ~22 layers in the 24-layer 0.5B vs ~60 in the 64-layer 32B), with intermediate layers defaulting to a Latin transliteration (Table 1). Given smaller models' lower strict script-following (0.641 for the 0.5B vs 0.831 for the 36-layer 3B), the authors propose a depth-bottleneck hypothesis, explicitly framed as correlational. Stated limitations: a single model family, 4-bit quantization of the 14B/32B checkpoints, exclusion of Armenian/Chinese/Japanese from the commitment analysis due to low success rates, and probing at a single token position.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Input script identity is linearly decodable from layer 1 at ~0.98 balanced accuracy in all six Qwen 2.5 sizes (§3).
- The requested output script is fully encoded (balanced accuracy 1.00) by layers 3–5, yet the output distribution only commits to non-Latin scripts in the final layers, above ~85% of model depth (Figs. 1, 3).
- Intermediate representations default to Latin, and the target script emerges as a transliteration of the held Latin prediction in the last 3–8 layers (Table 1, Fig. 4).
- Smaller models' weaker script-following (strict rates 0.641–0.831, Table 3) is consistent with insufficient depth to complete late script conversion rather than failure to encode the instruction — an explicitly correlational hypothesis (§5).
Strengths
Conceptually sharp three-way dissociation
Separating input script, requested-output script, and used-output script as distinct probe targets (§2.4) goes beyond prior work like RomanLens, which the paper correctly positions itself against (§1); the dissociation 'encoded early / realized late' is the paper's clearest contribution.
Convergent multi-method evidence
Probing (Figs. 1–2), logit-lens commitment layers (Fig. 3), the Table 1 layer-by-layer example (Latin 'capital' flipping to Cyrillic 'К' at layer 34 of 36 in the 3B), probability-mass curves (Fig. 4), and logit-margin analysis (Fig. 7, margin 0.5–5 rising sharply in final layers) all point the same direction, reducing single-method artifacts.
Non-monotonic depth observation is well exploited
Scale, determinism, and release
Weaknesses
Internally inconsistent probe methodology reporting
Quantization confound in cross-size claims
Depth-bottleneck claim is correlational with no intervention
Thin statistical reporting and one direct contradiction
Unnormalized script-mass comparison and survivorship conditioning
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks where, across transformer layers, script knowledge lives: recognition of the input script, encoding of the instructed output script, and commitment of the actual output distribution. Using six Qwen 2.5 Instruct checkpoints (0.5B–32B; 24–64 layers; Table 2), the authors claim a two-stage process — early encoding, late commitment — with intermediate representations defaulting to Latin, and propose a depth-bottleneck hypothesis for small models' weaker script-following. Two methods (probing, logit lens) do converge on the dissociation; the depth claim remains explicitly correlational (§5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'late-layer script commitment' in this paper mean?
The paper finds that although a language model encodes the requested output script within its first few layers, its output distribution keeps predicting Latin-script tokens for most of its depth and only switches to the requested script (e.g., Cyrillic, Arabic, Hebrew) in the final layers — typically above ~85% of model depth for non-Latin targets. The held Latin prediction is described as a transliteration of the intended answer.
Which models were studied, and do the findings generalize?
Six sizes of the Qwen 2.5 Instruct family (0.5B, 1.5B, 3B, 7B, 14B, 32B), spanning 24–64 layers; the two largest ran with 4-bit quantization. The authors test only this one family, so generalization to other architectures (Llama, Gemma, etc.) remains untested — though they cite RomanLens's related finding on different models as corroboration.
Do smaller models fail at non-Latin scripts because they don't understand the instruction?
No — probes show the instruction is encoded early even in small models. The authors argue, explicitly as a correlation rather than a proven cause, that small models lack enough layers to complete the late script-conversion step. Supporting evidence: the 36-layer 3B follows scripts better (0.831 strict rate) than the 28-layer 7B (0.799) and even the 64-layer 32B (0.754).
Is the paper's data or code available?
The prompt data (8,304 prompts over nine scripts and 17 language-to-script mappings) is released on GitHub at IDSIA-NLP/LLMScriptCommitment. All models are public Qwen checkpoints, experiments used greedy decoding on a single 46GB NVIDIA L40S GPU, so the pipeline is reproducible with moderate compute.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–8 are referenced but not analyzable from the extracted text; numeric ranges for probe peaks, commitment layers, and margins rely on the authors' prose descriptions rather than direct figure inspection.
- Appendix F/G breakdowns (per-script success, wrong-script distribution) are figures only; per-script confidence intervals could not be verified.
- The Appendix E layer-0 'chance' range (0.47–0.54) conflicts with §3's ~0.11 claim; flagged as likely reporting error rather than adjudicated.
- The mapping between 17 stated language-to-script mappings and the 8,304-prompt total cannot be fully reconciled from the extracted text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.