Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context
Nawar S. Alseelawi, Mustafa S. Aljumaily
cs.CL · cs.AI
Quick verdict
First Iraqi-Arabic LLM benchmark: rigorous 340-item dual-track design and a notable over-refusal finding, but half the axes lack results and counts conflict.
Summary
The authors introduce Mizan, presented as Iraq's national benchmark for evaluating LLMs on Iraqi Arabic, pairing an MSA baseline track with an Iraqi track across six axes: dialect comprehension, dialect generation, MSA-Iraqi translation, Iraq-specific knowledge, official-document extraction, and safety. All 340 items (265 Iraqi, 75 MSA) are natively authored and dually reviewed, with chi-square-audited answer positions and Wilson intervals on every score. A pilot evaluates 27 systems on the 190 auto-scored items; 150 generative items await human judging. Key results: the MSA track saturates (nine models at 100.0 in Table 1, though the prose says eight) while the Iraqi track spreads over 23 points, most models carrying a 14-18-point internal gap; document extraction caps all systems at 32-56.4; Arabic-specialized SILMA-9B (63.2) and ALLaM-7B (64.7) trail size-matched generalist ministral-8b (73.9), though both ran in 4-bit quantization; and the safety-hardened claude-fable-5 deterministically refuses innocuous Iraqi vocabulary items (reported as 12% of Iraqi comprehension) while scoring 100.0 on MSA. Main limitations: half the axes have no results yet, 50 items per axis yield roughly ±10-point intervals, the entire bank is public development tier with a sealed set deferred to v1.0, Mosuli Arabic is uncovered, no human baseline exists, and several counts (refusal tallies, perfect-MSA totals) are internally inconsistent.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Mizan's MSA track saturates (multiple systems at 100.0) while the Iraqi track discriminates, with most models carrying a 14-18-point internal gap between their two tracks.
- Official-document field extraction confines all 27 systems to 32-56.4, the one axis where the entire field, frontier models included, fails substantively rather than formatting-wise.
- Arabic-focused specialization currently behaves as MSA specialization: SILMA-9B (63.2) and ALLaM-7B (64.7) fall roughly ten points below the size-matched generalist ministral-8b (73.9) on the Iraqi track.
- The safety-hardened claude-fable-5 deterministically refuses innocuous Iraqi vocabulary items (reported as 12% of Iraqi comprehension) while scoring 100.0 on MSA — a dialect-triggered over-refusal mode no MSA benchmark can observe.
- Mizan is, to the authors' knowledge, the first comprehensive originally-authored evaluation framework dedicated to Iraqi Arabic and the Iraqi civic context.
Strengths
Native authorship with enforced item discipline
Every item is written natively in its target variety and dually reviewed (§3.1); §3.4 documents rejections for lexical rarity, ambiguous keys, and contested ground truth, preserving a two-key geography item (Tigris-Euphrates confluence at Qurna vs Karmat Ali) as a design lesson — a concrete answer to the translationese critique leveled at AraDiCE and DialectalArabicMMLU (§2.3).
Above-average statistical hygiene for a dialect benchmark
95% Wilson intervals on every published score; chi-square goodness-of-fit fails to reject uniform answer positions for every (track, axis) group (all ≤1.20, df=3, p>0.05, §3.1); two independent full rounds show mean |Δ| 0.5 and max ~1.2 points, all within intervals (§6.4); extraction's binomial treatment is explicitly disclosed as a conservative bound for [0,1] fractions (§5).
Novel over-refusal finding with official evidence
Integrity protocol exercised, not merely described
Candid limitations and deployment-realistic disclosure
Weaknesses
Half the benchmark has no results
Numeric inconsistencies around the flagship finding
Quantization confound in the 'specialization law'
Thin power and an 'equal standing' tension
Abstract quantifiers drift from the table
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Mizan (arXiv 2609.13980) introduces a dual-track benchmark for Iraqi Arabic: an MSA baseline track and an Iraqi track spanning six axes — dialect comprehension, dialect generation, bidirectional MSA-Iraqi translation, Iraq-specific knowledge, official-document field extraction, and safety. All 340 items (265 Iraqi / 75 MSA, §3.6) are natively authored and dually reviewed. A pilot runs 27 systems — from llama-3.1-8b to GPT-6 Astra, evaluated three days post-release — on the 190 auto-scored items; the 150 generative items await human judging. Four headline claims: (1) MSA saturation vs Iraqi discrimination, with a 14-18-point internal gap for most models; (2) a 'document wall' confining every system to 32-56.4; (3) 'Arabic specialization is MSA specialization'; (4) a dialect-triggered over-refusal mode in claude-fable-5 (§1). Claims are mostly hedged ('to our knowledge', intervals everywhere), but the abstract's 'consistent 14-18-point gap' overstates Table 1, where per-model gaps run from 9.5 (llama-4-maverick, 92.5→83.0) to 22.7 (claude-fable-5, 100.0→77.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Mizan benchmark?
Mizan is a 340-item benchmark for evaluating LLMs on Iraqi Arabic, pairing an MSA baseline track with an Iraqi track across six axes: dialect comprehension, dialect generation, MSA-Iraqi translation, Iraq-specific knowledge, official-document extraction, and safety. All items were originally written by native Iraqi speakers and dually reviewed, with statistically audited answer positions and Wilson confidence intervals on every published score. A 27-model pilot evaluation accompanies the release.
What is the dialect-triggered over-refusal finding?
The safety-hardened claude-fable-5 deterministically refused innocuous Iraqi-Arabic vocabulary questions (e.g., the meaning of 'khosh', meaning 'good'), returning an official API stop reason of 'refusal' with category 'cyber'. This affected a reported 12% of Iraqi comprehension items while the model answered the entire MSA track; none of the other 26 systems refused anything. The authors argue that safety alignment calibrated without a dialect can turn that dialect's ordinary vocabulary into blockable content.
Which models performed best on Mizan?
On the auto-scored pilot board, Gemini 2.5 Pro and GPT-5 tie at the top (90.9 overall, 84.9 Iraqi track), with GPT-6 Astra — evaluated three days after release — third at 90.7. Exact ranks in the leading cluster are statistically unsettled at the pilot sample size. All top systems score 100.0 on MSA but only 82-85 on Iraqi, and every system fails the document-extraction axis, capped between 32 and 56.4.
Is the Mizan dataset public?
Yes — the paper releases the platform code under Apache-2.0 and a public development set with a datasheet. The entire pilot bank is deliberately development-tier; a sealed private test tier is planned for version 1.0. Until then, leaderboard results are explicitly development-tier and could in principle be contaminated by models trained on the published items.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as captions in the extraction; per-model confidence-interval overlaps and the gap chart could not be independently checked.
- Generative axes (150 items: generation, translation, safety) have no results in this version, so the review covers only the five auto-scored axes.
- The numeric inconsistencies flagged (eight vs nine perfect MSA scores; 6/50 vs seven refused items) were verified against the embedded Table 1 and §7.1 prose, but HTML-to-text extraction could in principle have altered table cells.
- No appendix or datasheet content was present in the extraction; the claimed releases (code, dev set, certificates) could not be inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.