YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models
Mahmoud Reda, Salam Khalifa, Reham Marzouk, Nizar Habash
cs.CL
Quick verdict
Large 663K-entry Arabic morphological generation benchmark from CamelMorph; LLMs struggle on rare/cliticized forms; gold labels hinge on one resource.
Summary
The authors present YallaMorph, a benchmark of 663,804 entries for controlled Arabic morphological generation: given a lemma, POS, gloss, and feature bundle, a model must produce the inflected form or declare the configuration invalid. Built from CamelMorph MSA (36,101 lemmas), it samples lemmas across root class, frequency, paradigm completeness, and stem complexity into baseword (238,266) and cliticization (425,538) subsets; ~13% require a null (invalid) answer and 3.2% have multiple gold forms. Seven instruction-tuned LLMs were evaluated zero-shot and 10-shot under diacritized, undiacritized, and normalized scoring. GPT-5.4 leads (10-shot diacritized Any Match Accuracy 51.9; undiacritized 67.0), followed by Gemini 3.1 Flash-Lite (49.7 diacritized), while Arabic-centric open models (Jais-2, ALLaM, Fanar, Qwen) score 2.3–21.5. Accuracy tracks surface-form frequency (81.3 for the 10M+ band vs 52.3 for unseen forms; null configurations only 12.5). Error analysis of 200 GPT outputs shows diacritization and invalid-config detection dominate; cliticization mainly complicates host-word realization. Limitations: gold forms come from the same CAMeL Tools/CamelMorph generator that defines validity (multi-reference only 3.2%), lemma frequencies derive from a preliminary pipeline diverging from final BAREC-10M annotations, no statistical significance testing, and the cliticization subset excludes masculine/feminine-only paradigms, confounding cross-subset comparisons. Benchmark data and prompts are public on GitHub.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Controlled Arabic morphological generation remains difficult for current LLMs, especially for cliticized, unseen, and morphologically rare configurations.
- Arabic-centric open models (Jais-2, ALLaM, Fanar, Qwen2.5) substantially underperform GPT-5.4 and Gemini despite their Arabic specialization (Table 6: 2.3–21.5 vs 51.9 diacritized 10-shot AMA).
- Performance strongly tracks surface-form and specification frequency (Tables 8–9), indicating reliance on distributional exposure and memorization rather than robust morphological generalization.
- Cliticization is rarely the direct error source; it mainly increases the difficulty of realizing the host word correctly (Section 5.4).
- Models rarely detect morphologically invalid configurations, with null-reference accuracy of only 12.5% (Table 8).
Strengths
Scale with linguistically motivated stratification
663,804 entries (Table 4: 238,266 baseword + 425,538 cliticized) sampled from CamelMorph MSA's 36,101 lemmas across root class, frequency band, paradigm completeness, and stem complexity (Table 1), with explicit priority hierarchies to keep strata non-overlapping; includes invalid configurations (12.9%, Table 5) and multi-reference cases (3.2%).
Broad, well-instrumented evaluation matrix
7 instruction-tuned models × zero-shot/10-shot × English/Arabic prompts × diacritized/undiacritized/normalized scoring (Tables 6, 16, 17), plus per-dimension breakdowns (Table 7), surface-frequency bands (Table 8), and specification-frequency percentiles (Table 9).
Unusually honest error analysis
Cost and runtime transparency
Release readiness
Weaknesses
No uncertainty quantification anywhere
Gold-label circularity on a single resource
Confounded baseword-vs-clitics comparison
Frequency estimates from a preliminary pipeline
No fine-tuned baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
YallaMorph is a benchmark-and-evaluation paper. The authors construct a 663,804-entry testbed (Table 4: 238,266 baseword + 425,538 cliticized entries) for controlled Arabic morphological generation: given a lemma, POS, gloss, and feature bundle, a model outputs the inflected surface form or IMPOSSIBLE. The space is drawn from CamelMorph MSA's 36,101 lemmas (Table 2), whose exhaustive cliticized enumeration would reach ~4.44B forms, hence guided stratified sampling. Central claims: (1) controlled morphological generation remains hard even for frontier LLMs; (2) Arabic-specialized open models underperform proprietary multilingual systems; (3) performance is frequency-driven, evidencing memorization over systematic generalization. The evidence presented supports these claims, with the methodological caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is YallaMorph?
YallaMorph is a benchmark of 663,804 test entries that checks whether large language models can generate the correct Arabic word form when given a lemma, part of speech, gloss, and explicit morphological features — or correctly answer IMPOSSIBLE when the requested feature combination cannot exist in Arabic. It was built by NYU Abu Dhabi's CAMeL Lab from the CamelMorph MSA lexicon and is publicly downloadable on GitHub.
Which models performed best on YallaMorph?
GPT-5.4 was the strongest system, reaching 51.9% Any Match Accuracy in the strictest fully diacritized setting with 10-shot prompting (67.0% without diacritics). Gemini 3.1 Flash-Lite was second at 49.7%. Arabic-focused open models scored far lower: Fanar-2-27B, Jais-2-70B/8B, ALLaM-7B, and Qwen2.5-7B ranged from about 2.3% to 21.5% in the same setting.
Why is Arabic morphology hard for LLMs according to this paper?
The paper shows model accuracy collapses on forms that are rare or absent from training data (52.3% accuracy for unseen forms vs 81.3% for the most frequent band) and that models almost never recognize when a requested form is morphologically impossible (only 12.5% accuracy on such cases). This suggests models memorize frequent patterns rather than applying systematic morphological rules.
Does YallaMorph cover Arabic dialects?
No. The benchmark covers Modern Standard Arabic as encoded in the CamelMorph MSA resource. The authors list extension to Arabic dialect varieties and contextual (in-sentence) generation as future work, and caution against reading YallaMorph scores as a measure of overall Arabic language ability.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Arabic-script strings (Figure 1 glyphs, Table 3 example lemmas, and the Appendix E.2 Arabic prompt template) are garbled by PDF extraction; all numeric tables and English prose extracted intact, so no numeric claim in this review relies on the garbled portions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.