5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs
Md Mahir Jawad, Galib Mahmud Jim, Rafid Ahmed, Mir Sazzat Hossain, Md Fahim, Md Farhad Alam Bhuiyan
cs.CL
Quick verdict
发布包含 6,000 条目的 5 方言孟加拉语基准，涵盖转写；LoRA 优于闭源模型；转写惩罚控制良好且可信。
Summary
本文介绍了 5-Dialects-BN，这是一个包含 6,000 条目的基准，涵盖了五种地区性孟加拉方言（Chittagong 1,900 条；Noakhali 1,500 条；Sylhet 1,200 条；Barisal 700 条；Rangpur 700 条），每个条目包含五个对齐字段：原生文字方言文本、罗马化转写、标准孟加拉语翻译、英语翻译以及主观性标签。注释由 15 名语言学本科生制作，经过母语人士验证（约 24,000 个 harvested 候选词中拒绝了 34%），并在 1,500 个条目的双重注释子集上进行了验证（Cohen's κ 0.71–0.78；英语的 ROUGE-L 为 0.84）。作者在方言到英语的翻译、主观性分类和方言到标准规范化任务上对七个 LLM（Gemini 3 Flash、GPT-4o-mini、Claude Haiku 4.5；Qwen-3-4B、Gemma-4-4B、Llama-3.1-8B、Mistral-7B）进行了基准测试，采用了零样本、少样本、思维链和 LoRA 方案，每种方案都包含原生和罗马化脚本。主要发现有两个：每个方言仅使用 160 个样本进行 LoRA 微调，其表现优于所有闭源提示基线（Mistral-7B 73.6 BLEU，而 Gemini 零样本为 46.4）；罗马化输入在所有测试的模型-方案对中持续降低性能，通过往返、多方案和分词器控制，归因于多对一的音位丢失和子词碎片化。局限性包括单一的 LoRA 预算而没有扩展曲线、仅提示的规范化、没有人工评估，以及不一致的测试折叠大小报告。
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 5-Dialects-BN is the first multi-annotation Bangla dialect benchmark aligning Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five regional dialects (6,000 entries).
- LoRA fine-tuning of open-source LLMs on just 160 examples per dialect surpasses every closed-source zero-shot and few-shot baseline on translation and subjectivity classification (e.g., Mistral-7B 73.6 BLEU vs. Gemini 3 Flash 46.4 zero-shot).
- Romanized transliteration input consistently and severely degrades current LLM performance across all tested models and regimes, reversing in-context benefits reported for earlier smaller multilingual models.
- The transliteration penalty is attributable to many-to-one phonemic information loss (round-trip CER 0.14–0.21 recovers almost none of the gap) and atypical subword fragmentation, and holds across Avro, ITRANS, and ISO-15919 schemes.
Strengths
Factorial transliteration analysis with genuine causal controls
Section 5.2 isolates the mechanism three ways: a deterministic round-trip inverse-transliteration control (Table 15: Gemini zero-shot chrF++ native 62.13 vs round-trip 55.57 vs Romanized 57.81, CER 0.14–0.21, showing the penalty is irreversible); multi-scheme robustness over ITRANS/ISO-15919 (Table 16: Avro mildest at −4.63); and an item-level token-inflation regression (Table 14: β +7.86 to +60.92, p ≤ 0.035).
Self-critical baselines and failure diagnostics
A copy-input baseline (39.66 chrF++, Table 5) exposes Qwen-3-4B's 34.1% verbatim pass-through inflation; Llama-3.1-8B's 5.9% correct-to-unrelated flip rate is quantified as a 15–30× outlier (Table 11) and traced by a demonstration-count sweep (Table 12); Qwen's 91.4% zero-shot format collapse is separated from capability.
Documented annotation quality with a bias control
Honest scope statement
Weaknesses
Inconsistent test-set reporting
No uncertainty quantification on headline comparisons
Closed-model versioning and citation mismatches
LLM-co-produced references scored by LLM-era metrics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a resource-plus-evaluation benchmark paper. The authors release 5-Dialects-BN: 6,000 manually annotated utterances across five Bangla dialects (Chittagong 1,900; Noakhali 1,500; Sylhet 1,200; Barisal 700; Rangpur 700), each entry carrying five aligned fields — native-script dialect text, Romanized transliteration, Standard Bangla translation, English translation, and a binary subjectivity label. Around the resource they run a factorial study: seven LLMs × three tasks (dialect→English MT, subjectivity, dialect→Standard normalization) × four regimes (zero-shot, 36-demo few-shot, eight-step CoT, LoRA) × two input scripts. Three claims are advanced: the resource is the first multi-annotation Bangla dialect benchmark with transliteration; LoRA on 160 examples/dialect beats all closed-source prompting baselines; and Romanized input consistently degrades modern LLMs, reversing in-context benefits reported for older multilingual models. The 'first' positioning is argued via Table 6 against DIALTSA-BN (600 utterances, four dialects, same senior-author group), Vashantor, ONUBAD, ChatgaiyyaAlap, and ANCHOLIK-NER; the novelty claim is plausible but depends on those resources' exact field coverage.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 5-Dialects-BN?
A benchmark of 6,000 manually annotated utterances from five regional Bangla dialects (Chittagong, Noakhali, Sylhet, Barisal, Rangpur). Each entry aligns five fields: the dialectal text in Bangla script, a Romanized transliteration, a Standard Bangla translation, an English translation, and a subjective/objective label. It is released under CC-BY-SA-4.0 on Hugging Face and Kaggle.
Does transliteration hurt LLM performance on Bangla dialects?
According to the paper, yes: across seven LLMs, three tasks, and prompting plus LoRA regimes, Romanized input consistently scored below native Bangla script (native won in all 19 reported model-regime pairs). The authors attribute this to irreversible phonemic information loss — back-transliterating to native script (character error rate 0.14–0.21) recovered almost none of the gap — and to subword token fragmentation; the penalty held across Avro, ITRANS, and ISO-15919 schemes.
Can small fine-tuned models beat GPT or Gemini on Bangla dialect tasks?
On this benchmark, yes: LoRA fine-tuning of four open 4–8B models on only 160 training examples per dialect exceeded all closed-source zero-shot and few-shot prompting results on translation and subjectivity — for example, Mistral-7B reached 73.6 BLEU (COMET 89.97) versus Gemini 3 Flash's 46.4 BLEU zero-shot.
Which Bangla dialects does the dataset cover and how balanced is it?
Five dialects: Chittagong (1,900 entries), Noakhali (1,500), Sylhet (1,200), Barisal (700), and Rangpur (700). The imbalance reflects natural online availability; evaluation subsets are dialect-balanced, but the authors warn the skew may bias future cross-dialect transfer studies.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- One metric value appears lost to PDF extraction in appendix Table 21 (the Gemma-4-4B Noakhali row shows 7 of 8 values); all other tables extract cleanly.
- The abstract supplied in the task metadata ends mid-word ('parameter-ef'), but the embedded full text is complete through the references and appendices — no other truncation detected.
- Appendix Tables 21–26 extract as dense numeric blocks; per-cell row alignment was spot-checked against main-text aggregates, but deep per-dialect cells carry minor misalignment risk.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.