Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu
Farah Adeeba, Abdul Rafae Khan, Rajesh Bhatt, Hassan Sajjad
cs.CL · cs.AI · cs.LG
Quick verdict
发现 LLM 生成的乌尔都语故事虽然流畅，但在文化内涵上较为肤浅；尽管有强大的母语语料库支持，但因内部计数不一致和缺乏显著性检验而有所削弱。
Summary
本文测试了多语言大模型（LLM）是否能真正写出合格的乌尔都语叙事。在四个类别（翻译的英文经典、乌尔都语正典文学、寓言以及作者自创的中性控制组）的 31 个标题中，GPT-5.1、Qwen-3-Max 和 DeepSeek-3.1 各生成了一个故事，形成了一个包含 93 个故事的语料库（约 152,000 字）。一位乌尔都语母语人士根据 13 个标签的分类法对所有故事进行了标注；另外四位母语人士在 60 个故事的子集上验证了一致性（κ = 0.81）。标注得出了 1,675 个错误，其中 1,359 个是语言类错误（语法、性一致、拼写、间距）。标题重复率梯度——记忆深刻的寓言约为 0.06，英文经典约为 0.45，乌尔都语正典标题约为 0.58（峰值为 2.79）——表明缺乏源文本的模型会从字面上拆解标题。文化层面的失败包括亲属称谓混淆、穆斯林角色被火化、巴基斯坦背景下的西方物品以及 86 个字词的脚本污染；男性主角数量以 3.7:1 的比例超过女性主角。GPT-5.5 后期编辑步骤解决了约 90% 的表面错误，但仍存在文化和语篇层面的失败。局限性：规模适中，每种条件下仅生成一次，人工校准的记忆阈值，缺乏显著性检验，同族模型评判，以及若干内部数值不一致（例如，九个与十三个标签；9,460 与 11,597 个句子）。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Contemporary multilingual LLMs produce surface-fluent but linguistically fragile Urdu, with errors concentrated on features English lacks (ergative case, gender agreement, retroflex orthography).
- Generation behaviour tracks training-data coverage: memorised fables are reproduced, while Urdu-canon titles absent from training data trigger 'lexical title unpacking' (TRR ≈0.58, peak 2.79).
- Cultural grounding failures — kinship-term misuse, religious anachronisms such as a cremated Muslim character, Western object substitutions, and 86 script-contaminated words — persist under prompting.
- Generated stories show structural bias: male protagonists outnumber female 3.7:1, and positivity ratios of 0.78–0.82 mean most stories contain no negative relationship.
- A GPT-5.5 sentence-level post-editing pass fixes ~90% of surface linguistic errors but cannot repair cultural, discourse-level, or register failures such as DeepSeek's Punjabi switching.
Strengths
Credible native-speaker annotation with strong IAA
One native Urdu speaker labelled all 93 stories under a 13-label taxonomy (Table 2); four additional native speakers re-annotated a stratified 60-story subset with κ = 0.81 overall, stable per model (0.806–0.808, Table 19).
Four-way title design cleanly separates memorisation from competence
10 translated English classics, 11 Urdu-canon titles, 4 fables (plot-memorisation control) and 6 author-composed NUT titles (leakage control in the spirit of Gómez-Rodríguez & Williams 2023) let the authors disentangle language coverage from text coverage.
TRR is a judge-free, length-invariant diagnostic
Unusually thorough robustness battery
Qualitative analysis grounded in Urdu morphosyntax
Weaknesses
Multiple internal numerical inconsistencies
No inferential statistics
Same-family judging with material score shifts
Small, single-sample design with unexplained counts
Human baseline is not prompt-matched
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether LLMs marketed as multilingual can genuinely generate Urdu — a language with hundreds of millions of speakers but thin web-training coverage. Using story generation over 31 titles in four categories (10 TSST translated English classics, 11 UST Urdu-canon titles, 4 fables, 6 author-composed NUT controls), each issued once to GPT-5.1, Qwen-3-Max and DeepSeek-3.1, the authors build Urdu-Stories (93 stories, 151,965 words). Five claims: (1) surface fluency masks linguistic fragility concentrated on English-absent features (ergative *ne*, gender agreement, retroflex orthography); (2) behaviour tracks training coverage — a Title Repetition Rate gradient from ~0.06 (fables, plot memorised) through ~0.45 (English classics, theme remembered) to ~0.58 with peak 2.79 (Urdu canon, 'lexical title unpacking' — QWEN renders Chughtai's *Liḥāf*, a metaphor for hidden lesbian desire, as a literal quilt story repeating the word 35 times); (3) cultural grounding fails (kinship-term misuse, a Muslim character cremated, a 'gas station' in a Punjabi village, 86 script-contaminated words, DeepSeek switching into Punjabi); (4) structural bias: 3.7:1 male protagonists, positivity ratios 0.78–0.82 with most stories containing no negative relationship; (5) GPT-5.5 remediation fixes ~90% of surface errors but not cultural/discourse failures.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Urdu-Stories dataset and how big is it?
A corpus of 93 Urdu stories generated by three LLMs (GPT-5.1, Qwen-3-Max, DeepSeek-3.1) for 31 titles in four categories: translated English classics, canonical Urdu literature, traditional fables, and newly composed control titles. Stories were manually annotated by native Urdu speakers under a 13-label error taxonomy, with inter-annotator agreement κ = 0.81 on a 60-story subset. The authors state they release the data, annotations and evaluation scripts.
What are the main weaknesses of LLMs in Urdu found by this paper?
Stories are surface-fluent but problematic: 1,675 annotated errors (1,359 linguistic, covering grammar, gender agreement, spelling and spacing); cultural failures such as misused kinship terms, a Muslim character being cremated, Western objects like gas stations in Pakistani villages, and 86 script-contaminated words; male protagonists outnumbering female 3.7:1; and heavy repetition of the prompt title when the model does not know the source text.
What is Title Repetition Rate (TRR) in this paper?
The number of exact occurrences of the prompt title per 100 words of story body. It was near zero (~0.06) for well-known fables whose plots are memorised, intermediate (~0.45) for translated English classics, and highest (~0.58, peak 2.79) for Urdu-canon titles absent from training data — showing that models literally unpack unfamiliar metaphorical titles instead of recognising them.
Can prompt engineering or post-editing fix these Urdu generation errors?
Only partly. A GPT-5.5 post-editing pass on 15 stories fixed about 90% of surface linguistic errors (grammar, agreement, spelling, spacing), but could not repair cultural anachronisms, long-range gender inconsistencies, or DeepSeek's switching into Punjabi — errors requiring cultural knowledge and discourse-level state rather than sentence-level proofreading.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with explicit '[...sampling gap...]' markers (within §3.4 and around Appendix M); content in those gaps was not reviewable, though every number cited in this review appears in the visible text.
- Urdu script in the extraction is heavily garbled (many tokens render as unrelated Unicode blocks), so the linguistic correctness of specific example sentences could not be independently verified — only the paper's descriptions of them.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.