Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?
AbdulRahman A. Morsy, Aya Zirikly
cs.CL · cs.AI
Quick verdict
Claims first controlled LLM maqama-generation study; eval pipeline credible, but Table 1 contradicts Tables 2/3/10 and abstract overstates zero-shot results.
Summary
The paper presents the first controlled evaluation of whether large language models can generate classical Arabic maqamat — rhymed-prose (saj') narratives with dense rhetoric and episodic structure. Five models (GPT-4o, GPT-5.4-mini, ALLaM-7B-Instruct, Qwen3-8B, LLaMA-3-8B-Instruct) generate 1,050 texts (14 topics × 3 prompting regimes × 5 samples), judged by Claude Sonnet 4.5 through absolute 1-5 rubric scoring and pairwise comparison, corroborated by a second judge (Gemini 3.5 Flash), two expert human annotators (30 pairs, κ=0.9231), and non-LLM saj' proxies. GPT-5.4-mini dominates (Elo 1980.4; perfect pairwise win rate; mean Overall 4.17 vs 2.65 for GPT-4o), while open-weight models trail far behind; Arabic specialization (ALLaM) helps but is insufficient. Few-shot prompting most consistently improves saj' (Δ=0.429) and diacritization, whereas rhetoric and creativity are largely model-bound. Wilcoxon tests with Holm correction confirm all rankings except LLaMA vs Qwen. The authors state limitations of short-form generations, only 14 topics, and a small human study. This review additionally notes internal inconsistencies: Table 1 conflicts with Tables 2/3/10, and the abstract's claim that zero-shot yields the highest aggregate scores across all five models is not supported by Table 3.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- This is the first controlled evaluation of maqama generation, spanning five LLMs and three prompting regimes.
- GPT-5.4-mini dominates all systems (Elo 1980.4, +341 over GPT-4o, perfect pairwise win rate), while open-weight models, including Arabic-specialized ALLaM, remain far behind.
- Few-shot prompting most consistently improves saj' (Δ=0.429, Table 11) and diacritization, whereas rhetoric and creativity are largely model-bound (Δ≤0.066).
- The model ranking is robust across two LLM judges, human experts, statistical testing, and evaluation protocols, with LLaMA vs Qwen the sole unresolved pair.
Strengths
Multi-channel triangulation of evaluation
Absolute + pairwise LLM judging (Claude Sonnet 4.5, temperature 0, randomized pair order, §4.2) is corroborated by an independent second judge (Gemini 3.5 Flash: Spearman ρ=0.878 Overall, pairwise κ=0.907, §5.4), two expert human annotators (κ=0.9231, §5.5), and deterministic non-LLM saj' proxies (§5.6); absolute-vs-pairwise rank correlation is 1.00 (Table 8).
Statistical rigor with honest negatives
Wilcoxon signed-rank over all 1,050 matched judgments, Holm-Bonferroni across ten model pairs, bootstrap CIs on win rates (§5.4), and an explicitly flagged non-significant pair (LLaMA vs Qwen: 52.4% win rate, 95% CI [0.381, 0.667], §5.3) rather than burying it.
First controlled study of an unstudied genre
Behavioral findings beyond model rankings
Weaknesses
Internally contradictory Table 1
Abstract contradicts its own result tables
Grammar ceiling for the top model
Small, saj'-framed human evaluation
Underspecified experimental details
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims the first controlled evaluation of maqama generation by LLMs, framing the genre — rhymed prose (saj'), dense rhetorical ornamentation, episodic narration (citing Drory 2000; Hämeen-Anttila 2002) — as a stress test of literary competence beyond surface fluency. Four RQs cover feasibility (RQ1), prompting effects (RQ2), dimension-level prompt sensitivity (RQ3), and rubric-vs-pairwise agreement (RQ4). Central claims: GPT-5.4-mini dominates all systems; few-shot prompting most reliably improves saj'; rhetoric and creativity are model-bound; the ranking is robust across judges, protocols, and statistical tests. The novelty claim is plausible given the related-work section (poetry-focused prior work such as Sadallah et al. 2026 and Al Ghallabi et al. 2025), though 'first' claims always merit a literature sweep.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which LLM is best at writing classical Arabic maqama, according to this paper?
Per the paper's dual LLM judges (Claude Sonnet 4.5 and Gemini 3.5 Flash) and its two expert human annotators, GPT-5.4-mini ranks first (Elo 1980.4, winning all of its pairwise comparisons), followed by GPT-4o (Elo 1638.93). The open-weight models tested — ALLaM-7B-Instruct, Qwen3-8B, and LLaMA-3-8B-Instruct — scored far lower, with ALLaM best among them.
What is a maqama and why is it hard for language models?
The maqama is a classical Arabic prose genre from the 10th century (pioneered by al-Hamadhani, canonized by al-Hariri) built on rhymed prose called saj', dense rhetorical devices, and episodic storytelling. This paper argues it is hard for LLMs because it demands phonological control of rhyme, sustained classical register, and narrative coherence simultaneously — beyond the surface fluency models achieve in Modern Standard Arabic.
Does few-shot prompting improve maqama generation?
Partially. In the paper, few-shot prompting most consistently improves saj' (rhymed-prose density, Δ=0.429) and produces far more diacritized text (e.g., Qwen rose from 30.6 to 77.3 diacritic marks per 100 letters), apparently by imitation of the exemplars. However, effects on rhetoric and coherence vary by model, and human annotators preferred few-shot and rule-based outputs over zero-shot ones.
Is the paper's data and code available?
The authors state they release the full set of topics, prompts, and the 1,050 generated maqama outputs as supplementary material for reproducibility. A code release is not mentioned in the provided text, and the results depend on closed-source API models (GPT-4o, GPT-5.4-mini) and closed judges (Claude Sonnet 4.5, Gemini 3.5 Flash), so exact re-running is version-sensitive.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Arabic strings (Appendix A excerpt, Appendix B/C prompts) extracted as mojibake; judgments about prompt content rest on the supplied English translations only.
- Abstract is truncated mid-sentence in the extraction; figures and any remaining appendix content were not visible.
- The Table 1 inconsistency finding is based on the extracted text; the full PDF could reveal a normalization difference, though the captions as given conflict.
- Model and judge identities (GPT-5.4-mini, Gemini 3.5 Flash, Claude Sonnet 4.5) are taken as printed and were not independently verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.