Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations
Karthika Nhayakkat, Rajat Verma, Maharaj Brahma, Vetcha Gnana Mahesh, Maunendra Sankar Desarkar, Ganesh Ramakrishnan, Rohit Saluja
cs.CL
Quick verdict
Reordering/voice perturbations degrade 6 LLMs' Hindi/Malayalam math reasoning; credible benchmark with patching, but Malayalam quality evidence is thin.
Summary
The paper introduces IndicReStruct, a semantics-preserving structural-perturbation benchmark built from GSM8K's Hindi and Malayalam translations, with two variants: GSM8K-Reordered (constrained reordering of Samanvaya-style word groups under conjunction/punctuation boundary constraints, five generation seeds) and GSM8K-Voice (active-passive conversion), all produced with Gemini-2.5-Pro and quality-checked via embedding-similarity comparisons against random reordering, an LLM judge, and a 60-sample human check. Across six open LLMs (Gemma-2-9B/27B-it, GPT-OSS-20B, Llama-3.1-8B-it, Param-2-17B-A2.4B, Qwen3-30B-A3B) and four prompting strategies, reordering consistently degrades accuracy — e.g., Gemma-2-9B-it Hindi zero-shot CoT falls from 76.30 to 61.78, and Param-2-17B drops 31.21 points — with smaller but consistent drops under voice change (up to 29.41 points) and on Hindi ARC-Challenge. LoRA/DoRA finetuning on perturbed data (41.59/46.13) fails to reach the zero-shot reordered baseline (61.78). An error taxonomy attributes most failures to entity-quantity misalignment, and residual-stream activation patching on Gemma-2-9B-it (65 Hindi, 67 Malayalam items) finds intermediate layers (~5-25) most restorative, with entity-quantity tokens yielding the highest recovery. Limitations include the absence of an English or paraphrase control, thin Malayalam semantic-preservation evidence, LLM-judged quality with 9.4% rejects, and no significance testing.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Constrained, semantics-preserving constituent reordering causes consistent accuracy degradation across six LLMs, four prompting strategies, and both Hindi and Malayalam (e.g., Gemma-2-9B-it Hindi zero-shot CoT: 76.30 to 61.78).
- Active-passive voice transformation produces smaller but consistent drops, up to 29.41 points for Llama-3.1-8B-it on Hindi (Table 4).
- Reasoning failures under reordering are dominated by entity-quantity misalignment, and activation patching shows entity-quantity token representations restore correct answers at the highest rate.
- Intermediate transformer layers (~5-25) contribute most strongly to reasoning restoration under residual-stream patching on Gemma-2-9B-it.
- Neither prompting nor lightweight finetuning (LoRA 41.59 / DoRA 46.13 vs zero-shot 61.78 on Hindi reordered) closes the robustness gap.
Strengths
Linguistically grounded perturbation design
§3.2 permutes Samanvaya-style semantically cohesive word groups under protected conjunction/punctuation boundaries (Appendix A), explicitly avoiding meaning-destroying scrambles like (जा रहा है)→(रहा जा है), rather than naive token shuffling.
Breadth and internal consistency of the robustness result
Table 2 covers 6 models × 4 prompting strategies × 2 languages × 5 dataset-generation seeds with mean ± std; drops range from ↓3.66 (Qwen3-30B-A3B, Hindi zero-shot) to ↓31.21 (Param-2-17B-A2.4B, Hindi zero-shot), and the direction is uniform across all 24 cells.
Triangulated failure analysis
Honest negative finetuning result
Strong reproducibility posture
Weaknesses
No English or paraphrase control condition
Weak Malayalam semantic-preservation evidence
No inferential statistics; some effects within noise
Interpretability sample is tiny and lacks a null control
Generator-as-judge circularity in quality control
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether multilingual LLMs stay accurate when inputs change surface syntax but not meaning — a live question for free-word-order languages like Hindi and Malayalam. It introduces IndicReStruct (1,319 GSM8K-derived items per language) with two perturbations: constrained word-group reordering (GSM8K-Reordered, five generation seeds) and active-passive voice conversion (GSM8K-Voice), both LLM-generated with Gemini-2.5-Pro. Central claims: (1) consistent, large accuracy drops across six open LLMs and four prompting strategies; (2) failures concentrate in entity-quantity misalignment; (3) intermediate layers (~5-25) are most restorative under residual-stream activation patching; (4) lightweight finetuning cannot fix the gap. Scope is deliberately limited to two Indic languages and math reasoning (plus a Hindi ARC-Challenge extension), which the Limitations section states honestly.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the IndicReStruct benchmark?
IndicReStruct is a benchmark of 1,319 GSM8K math word problems in Hindi and Malayalam, each rewritten in two semantics-preserving ways: GSM8K-Reordered (constituent word groups scrambled under linguistic constraints) and GSM8K-Voice (active-passive voice conversion). It was built with Gemini-2.5-Pro and released on Hugging Face (karthika95/IndicReStruct).
Do LLMs get worse when Hindi or Malayalam word order is scrambled?
Yes, according to this paper. Across six open models and four prompting strategies, accuracy consistently dropped under reordering — for example Gemma-2-9B-it fell from 76.30% to 61.78% on Hindi zero-shot CoT, and the largest drop was 31.21 points (Param-2-17B-A2.4B, Hindi). Voice changes also hurt, with drops up to 29.41 points.
What causes the failures according to the paper?
An error taxonomy over the models' chain-of-thought finds entity-quantity misalignment the dominant failure mode, and activation-patching experiments on Gemma-2-9B-it show that restoring entity-quantity token representations recovers correct answers most often, with intermediate layers (~5-25) most restorative.
Can fine-tuning fix the sensitivity to word order?
Not in this paper: LoRA (41.59%) and DoRA (46.13%) trained on perturbed Hindi data both underperform plain zero-shot prompting on reordered questions (61.78%) for Gemma-2-9B-it, suggesting lightweight adaptation is insufficient.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Devanagari and Malayalam strings are heavily garbled by PDF extraction (matras rendered as spaces or substitution boxes), so linguistic judgments about specific reordered examples (Figure 1, Table 8) could not be independently verified from the text.
- Figures 2-5 carry no extractable numeric data; the error-category distribution and per-layer restoration values are taken from the prose claims only, not from figure contents.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.