Machine Translation between English and Syriac (East Syriac Dialect) using Statistical Machine Learning
Hadiana Sliwa, Hossein Hassani
cs.CL
Quick verdict
First English–Syriac SMT model + 38.8k Bible corpus (BLEU 23.54): valuable resource contribution; significance testing and corpus arithmetic are weak points.
Summary
The paper presents what the authors describe as the first phrase-based statistical MT system and parallel corpus for English-Syriac (Assyrian, East Syriac Madnkhaya script), an endangered language largely absent from NLP. The authors assembled 38,847 verse-aligned sentence pairs from the complete English-Syriac Bible: the New Testament from a pre-existing aligned dataset, the Old Testament built from scratch via PDF extraction, chapter:verse anchor segmentation, and manual alignment review by three bilingual annotators. Preprocessing comprises cleaning, Syriac devocalization (diacritic removal by Unicode code points), and BPE tokenization. Using Moses, GIZA++, and KenLM, six configurations were trained over four random train/test splits with a fixed 1,000-sentence MERT tuning set. Word-level BLEU falls monotonically from 22.54 (95/5 split) to 17.81 (70/30); the best configuration adds a 5-gram language model and an Operation Sequence Model, reaching 23.54. Human evaluation of 20 unseen sentences by 11 native speakers gives mean adequacy 3.42 and fluency 3.34 on 5-point scales. Fine-tuned mBART-50 and NLLB-200 produced empty output because Syriac is absent from their vocabularies. Main limitations: Bible-only training domain, single-reference BLEU, no statistical significance testing, and human evaluation restricted to the best model; corpora and scripts are stated to be publicly released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- This is the first phrase-based SMT model and the first systematically curated parallel dataset (38,847 verse-aligned Bible pairs) for English-Syriac (Assyrian), with corpora, scripts, and model publicly released.
- Devocalization plus BPE preprocessing makes Moses SMT viable for this pair; the best configuration (5-gram KenLM plus Operation Sequence Model, 95/5 split) reaches word-level BLEU 23.54.
- Training-data volume is the dominant factor on quality (BLEU 22.54 to 17.81 from 95/5 to 70/30), and the OSM's +1.00 gain addresses the English SVO-to-Syriac VS reordering divergence.
- Human evaluation by 11 native speakers on 20 unseen sentences yields mean adequacy 3.42 and fluency 3.34 out of 5, consistent with comparable low-resource Semitic SMT baselines.
- Out-of-the-box multilingual NMT is currently inapplicable because Syriac is absent from mBART-50 and NLLB-200 vocabularies, so their fine-tuned models emit empty output.
Strengths
First substantial public resource for an unstudied language pair
38,847 verse-aligned English-Syriac pairs from the complete Bible, with the ~23,000-verse Old Testament built from scratch via verse-number anchor segmentation and three-annotator manual review (Sections 5.1-5.2); corpora, scripts, and trained model are stated to be publicly released - a contribution that outlives the model itself.
Linguistically motivated preprocessing
Devocalization is justified by native Syriac writing practice (Section 2) and framed as reducing surface-form sparsity, consistent with standard Arabic/Hebrew MT preprocessing (Habash and Sadat 2006); BPE is a reasonable fit for root-and-pattern morphology (Section 4.2).
Systematic, well-documented ablation
Candid limitations and honest negative result
Weaknesses
No statistical significance testing anywhere
Corpus-size arithmetic does not reconcile
Cross-study comparison table mixes BLEU scales
Evaluation confined to one domain, one reference, one model
Failed NMT run substitutes for a neural baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What BLEU score did the English-Syriac machine translation model achieve?
The best configuration achieved a word-level BLEU of 23.54 (Moses phrase-based SMT with a 5-gram KenLM language model and an Operation Sequence Model, trained on a 95/5 data split). Baseline configurations scored between 17.81 and 22.54 depending on training-set size. In human evaluation, 11 native Assyrian speakers rated translations at 3.42/5 for adequacy and 3.34/5 for fluency on average.
Is there a parallel corpus for Assyrian (Syriac) machine translation?
Yes - this paper releases what the authors describe as the first systematically curated English-Syriac parallel dataset: 38,847 verse-aligned sentence pairs covering the complete Bible (Peshitta tradition), including an Old Testament portion built from scratch via PDF extraction and manual alignment review. The authors state the corpus, scripts, and trained model are publicly available; a separate secular corpus from the Kurdistan Region Ministry of Education is shared privately on request for academic use.
Why use statistical (Moses) machine translation instead of neural MT for Syriac?
With only about 39,000 training pairs, neural models lack sufficient data, and Classical Syriac script is absent from the vocabularies of multilingual models like mBART-50 and NLLB-200 - the authors' fine-tuned versions produced empty output for this reason. Prior low-resource studies (e.g., Aramaic-Hebrew Bible translation) also found SMT outperforming NMT at this data scale, which is why the authors chose the Moses phrase-based SMT framework.
Can this model translate modern, everyday Assyrian?
No. The model was trained exclusively on Biblical text, so its scores reflect a narrow liturgical domain. The authors explicitly warn that it should not be interpreted as a general-purpose translator: modern vernacular Assyrian, secular vocabulary, and non-liturgical discourse are outside its training data, though secular government documents are being processed for future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 and Syriac-script content (Table 1, Figure 2 sample) are garbled or absent in the extracted text; the claimed bimodal rating distribution (Figure 4) could not be verified.
- The 'Dataset Availability: available here' hyperlink target is not visible in the extraction; public availability of the corpus, scripts, and model is taken from the paper's statements only.
- Section 5.3 (Experimental Setup) is a single sentence in the extraction and may be truncated; bibliography errors (Guellil et al. dated 2007, duplicate Naaijer/van Peursen entries) could partly be extraction artifacts.
- The abstract in the metadata is truncated mid-sentence; all numbers cited in this review were cross-checked against the full embedded text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.