Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMs
Mai Mohamed Eida, Ryan Dolan, Paul de Nijs, Jonathan Dunn
cs.CL
Quick verdict
This paper argues LLMs prefer Cairene over Sa'idi Arabic: upstream minimal-pair evidence is convincing; the downstream degradation is smaller and patchier.
Summary
The paper asks whether large language models share the social prestige hierarchy between Cairene (CEA) and Sa'idi (SEA) Egyptian Arabic. From an SEA literary corpus, the authors build 134 minimal pairs by switching one phonetic, morphological, orthographic, or lexical feature to its CEA equivalent, plus a nonce ungrammatical control, and rank sentence probabilities across five open models (Jasmine-350M, Jais-2-8B, Jais-1-13B, Llama-3.1-8B, NileChat-4B). CEA is ranked first roughly 80% of the time everywhere except Llama-3.1-8B (50%), and SEA beats the ungrammatical control in about 90% of second choices, so models treat SEA as a legitimate but dispreferred variety. Downstream, three Arabic MMLU benchmarks are perturbed with SEA, CEA, and ungrammatical features; overall accuracy gaps between SEA and CEA span only 0.1 to 3.9 percentage points with one slight reversal, but on items answered correctly under CEA, drops reach 8.3 points for Llama-3.1-8B and 30-36 points for NileChat-4B. McNemar tests identify Jais-2-8B as having the most consistent downstream bias and Llama-3.1-8B the weakest. Main limitations: only 134 pairs from a literary corpus, reliance on older dialect surveys, and use of relatively small open-source models.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across five open LLMs, minimal-pair comparisons consistently rank CEA sentences as more probable than SEA equivalents, with the preference holding even when the surrounding context is otherwise SEA.
- Models treat SEA as a legitimate but dispreferred variety rather than as ungrammatical: SEA is ranked above a nonce-form control in roughly 90% of second-choice decisions.
- Perturbing Arabic MMLU benchmarks with SEA features degrades accuracy relative to CEA, but overall effects are small (roughly 0.1-3.9 points) and statistically scattered; drops are far larger on items answered correctly in CEA (up to 8.3 points for Llama-3.1-8B, 30-36 points for NileChat-4B).
- The downstream effect is amplified when multiple SEA features occur together, but its size and statistical significance vary substantially across models, so upstream preference strength does not translate proportionally into downstream harm.
Strengths
Well-motivated upstream design with a proper ungrammatical control
Minimal pairs (n=134, §3.1) are built from SEA-corpus sentences by altering one SEA feature at a time, with a nonce-form UG baseline matched in length and script. The key interpretive result — SEA ranked second over UG in ~90% of cases (Fig. 3) — cleanly separates 'dispreferred variety' from 'viewed as ungrammatical'.
Model coverage spanning four training regimes
Five models cover monolingual Arabic (Jasmine-350M), Arabic-English bilingual (Jais-1-13B/2-8B), multilingual (Llama-3.1-8B), and Egyptian-Arabic-continuing-pretrained (NileChat-4B), and the divergent downstream patterns across them are explicitly analyzed (§5) rather than hidden in an average.
Honest nuance in downstream interpretation
Striking subset effect sizes
Weaknesses
The abstract's headline claim overreaches the aggregate results
The 'SEA midpoint' conclusion is supported statistically for one of five models
Unreferenced statistics and no multiple-comparison control
High-frequency features blur 'dialect' and 'string noise'
Extensive typographic defects undermine confidence in the tables
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Some Dialects Are More Equal Than Others (arXiv:2609.23955)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs treat less prestigious Arabic dialects as ungrammatical?
Not exactly. In this study, five open-source models ranked sentences written in Sa'idi Egyptian Arabic (a lower-prestige rural variety) above a deliberately ungrammatical nonce version in about 90% of second-choice decisions, so models clearly recognize it as valid Arabic. However, they still ranked the prestigious Cairene equivalent first about 80% of the time, showing a preference rather than a grammaticality judgment.
How much does Sa'idi dialect wording hurt LLM accuracy on knowledge benchmarks?
Overall accuracy drops were small, between roughly 0.1 and 3.9 percentage points versus Cairene wording across three Arabic MMLU benchmarks, with one case slightly favoring Sa'idi. The effect became much larger on questions the model had answered correctly in Cairene: up to 8.3 points for Llama-3.1-8B and about 30-36 points for NileChat-4B when the same question was rewritten with Sa'idi features.
Which models were tested and how big are they?
The paper tested five open-source models between 350M and 13B parameters: Jasmine-350M (Arabic monolingual), Jais-2-8B and Jais-1-13B (Arabic-English bilingual), Llama-3.1-8B (multilingual), and NileChat-4B (continuing-pretrained on Egyptian Arabic). Three of these were used for the downstream benchmark experiments. No proprietary frontier models were evaluated.
Is the paper's code and data available?
Yes. A footnote in the introduction states that all data, code, and prompts are available at github.com/maimm2/EA_Dialectal_Bias, which supports reproducing the minimal-pair construction and the benchmark perturbation pipeline, though re-running inference on the 8B-class models needs a GPU.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Arabic script in Tables 3-4 extracted as mojibake, so the actual minimal-pair and feature-variant strings could not be verified from the text.
- Figures 1-10 were not analyzable (captions only); McNemar results exist only as narrative, with no statistics in the extracted text.
- Experiment 1 column heads and per-condition pairing are complex and were reconstructed from dense text; individual numbers were cross-checked but column alignment may hide errors.
- Item-count discrepancies (ArabicMMLU n=9,689 vs n=2,194 between single and mixed conditions) are unexplained in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.