Measuring the Cost of Variety Conflation in Multilingual MT Evaluation: Adding Mozambican Xichangana, Nyanja and Sena to FLORES+
Felermino D. M. A. Ali, Delfina Lázaro Mateus, Manuel Valente Mangue
cs.CL
Quick verdict
FLORES+ sets for three Mozambican Bantu varieties show sibling references inflate MT scores; controlled evidence is convincing, fine-tuning recovers most.
Summary
This benchmark/empirical paper extends FLORES+ with Portuguese-source evaluation sets for three Mozambican Bantu varieties: Xichangana, Mozambican Nyanja, and Sena, translated by native-speaker trained linguists via a custom CAT platform. Comparing each new reference with the existing FLORES sibling (Tsonga, Chichewa), and scoring fixed system outputs (NLLB-200, Google Translate, GPT) against both references, the authors show sibling references substantially overestimate quality: on devtest, substituting Xichangana for Tsonga cuts NLLB-200 by 13.10 and Google by 15.30 spBLEU points; on matched Nyanja subsets, replacing Chichewa with Mozambican Nyanja reduces scores by 3.03 and 6.10 spBLEU. Variant-aware fine-tuning of NLLB-200 (new regional tokens plus target markers) improves the intended targets by 7.04 spBLEU (Xichangana) and 5.33 (Mozambican Nyanja) on devtest while degrading sibling-language performance, and establishes a first Sena result of 12.64 spBLEU / 36.21 chrF++ on devtest; GPT is competitive on Tsonga and Chichewa but weak on the Mozambican varieties. Limitations acknowledged include Portuguese pivoting, and Sena lacks a comparison against the original NLLB baseline; the Mozambican Nyanja training set is comparatively small (~8,000 rows).
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sibling references substantially overestimate translation quality for the Mozambican varieties: on devtest, replacing Tsonga with Xichangana under identical hypotheses cuts NLLB-200 by 13.10 and Google by 15.30 spBLEU points.
- The Chichewa-to-Mozambican-Nyanja gap on matched devtest subsets is smaller but systematic: 3.03 spBLEU for NLLB-200 and 6.10 for Google.
- Variant-aware fine-tuning of NLLB-200 improves the intended targets (+7.04 spBLEU Xichangana, +5.33 Mozambican Nyanja on devtest) while degrading sibling-language scores, demonstrating controllable specialization.
- GPT is competitive on Tsonga and Chichewa but substantially weaker on the Mozambican varieties, showing zero-shot variety naming in prompts gives insufficient control.
- The finetuned model establishes a first reproducible Sena result of 12.64 spBLEU / 36.21 chrF++ on devtest, though not a demonstrated improvement over the original NLLB baseline.
Strengths
Controlled reference-swap design cleanly isolates the variety effect
Because FLORES+ is multi-parallel and the Portuguese source stays sentence-aligned (Sec. 4.1), off-the-shelf NLLB-200 and Google produce identical hypotheses within each sibling–Mozambican pair (Sec. 6.3), so score deltas attribute to the reference alone — e.g., NLLB-200 devtest 18.91/44.82 vs Tsonga vs 5.81/32.98 vs Xichangana (Sec. 7.1).
Genuine data contribution with credible provenance
First FLORES+ set for Sena (~2M speakers, Sec. 3); translators are native trained linguists resident where the variety is spoken, paid at/above local rates (Sec. 4.2), with a full translation–review–return workflow (Sec. 4.3, App. C CATfrica) following FLORES+ contribution guidelines.
Reference diagnostics substantiate the score gaps
Fine-tuning experiment shows the gap is addressable and target-specific
Honest reporting on weak spots
Weaknesses
No statistical uncertainty quantification
Sena lacks any baseline against the original model
'Matched Nyanja subsets' are never defined
Undiscussed domain skew in fine-tuning data
No human evaluation despite usability framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Measuring the Cost of Variety Conflation in Multilingual MT Evaluation
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper contribute?
It adds new FLORES+ machine-translation evaluation sets — dev and devtest translated from Portuguese — for three Mozambican Bantu varieties (Xichangana, Mozambican Nyanja, and Sena), created by native-speaker trained linguists, and shows that the existing FLORES Tsonga and Chichewa references substantially overestimate translation quality for the Mozambican varieties.
How much do MT scores drop when the Mozambican reference is used instead of the sibling reference?
On the devtest split, holding system outputs fixed, switching the reference from Tsonga to Xichangana lowers NLLB-200 by 13.10 and Google Translate by 15.30 spBLEU points; on matched Nyanja subsets, replacing Chichewa with Mozambican Nyanja lowers scores by 3.03 (NLLB-200) and 6.10 (Google) spBLEU.
Is Xichangana more different from Tsonga than Mozambican Nyanja is from Chichewa?
Yes. The paper's reference diagnostics give Xichangana/Tsonga 48.94% mean character similarity and 9.47% token overlap, versus 54.39% and 14.33% for Mozambican Nyanja/Chichewa, and the score drops are correspondingly larger for Xichangana.
Where can I download the FLORES Mozambique dataset?
The paper states the data is publicly available on Hugging Face at https://huggingface.co/datasets/MOZNLP/FLORES_MOZ.
Do the findings apply to other cross-border languages?
The authors argue the pattern — one language code covering national varieties with different borrowing sources and orthography — generalizes to other cross-border languages, and recommend variety-specific references, language identifiers, and disaggregated reporting; the paper itself only demonstrates the effect for the three Mozambican varieties.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The 'matched Nyanja subset' size and selection rule are not stated anywhere in the provided text, so subset-level numbers could not be independently recomputed.
- The GPT system is identified only by the API label 'gpt-5.6-sol' (queried September 2026) with no decoding parameters given, limiting GPT-side reproducibility assessment.
- Code availability is limited to what the paper states: the Hugging Face dataset and the CATfrica tool URL; no training/scoring code release is mentioned in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.