The University of Melbourne WMT 2026 CreoleMT Submission: A Domain-Balanced Approach to Low-Resource Pacific Creole Machine Translation
Raphaël Merx, Nick Thieberger, Ekaterina Vylomova
cs.CL
Quick verdict
WMT26 entry: domain-balanced NLLB fine-tuning beats open baselines by 3+ chrF++ on three Pacific creoles, but pis/bis scores use LLM-generated references.
Summary
This is the University of Melbourne's submission to the WMT26 Creole shared task: machine translation for three Pacific creoles — Tok Pisin, Bislama, and Solomon Pijin. The authors fine-tune NLLB-3.3B in two stages: broad pre-training on ~596k mostly religious parallel pairs, then continued fine-tuning on a domain-balanced mix augmented with back-translated news, spoken transcripts, and 6k pairs distilled from Gemini 3.5 Flash. One joint model per direction covers all three languages. On Bouquet Tok Pisin and novel human-original spoken-transcript test sets, the system beats all open baselines (NLLB-3.3B, Kreyòl-MT, MADLAD-400, CreoleM2M) by 3+ chrF++ in every direction and roughly matches Gemini 3.5 Flash (51.5 vs 51.9 on Bouquet; 47.7 vs 41.0 on transcripts), trailing on FLORES (47.6 vs 50.2). Single-factor ablations attribute the largest gains to loanword respelling (+7.7) and Gemini distillation (+2.4, nearly saturated within 1.5k pairs). Main limitations: the Bouquet references for Bislama and Pijin are Gemini-generated silver data sharing provenance with the distillation source; significance testing is sparse; and no code release is stated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Continued fine-tuning of NLLB-3.3B on a domain-balanced mix plus spoken transcripts and limited Gemini distillation beats every open baseline by at least 3 chrF++ in all directions with human-original references (Bouquet tpi, spoken transcripts).
- The shipped model matches Gemini 3.5 Flash on Tok Pisin Bouquet (51.5 vs 51.9) and exceeds it on spoken transcripts (47.7 vs 41.0), remaining 2.6 points behind on FLORES (47.6 vs 50.2).
- Distillation gains saturate early: the first 1.5k Gemini-distilled pairs give +1.54 chrF++ of the +2.73 total at 16k pairs.
- Loanword respelling alone adds +7.7 chrF++ for Pijin, showing chrF++ partly measures spelling conformity to the reference rather than translation quality.
- Joint training and domain-diverse data transfer across the three creoles, helping the two lowest-resource languages most (+1.8 bis, +1.4 pis, +0.7 tpi, Table 4).
Strengths
Honest evaluation provenance
Gemini-generated Bouquet references for Pijin/Bislama are explicitly marked as silver in Table 3 and Section 3.2 ('we mark those cells... and do not read them as gold'), and the conclusion calls them 'unsatisfactory' — rare candor.
Explicit, quantified decontamination
Section 3.1.1 reports pre-filter overlap with the shared-task eval (75% tpi, 95% bis, 99% pis on the English side; 66% tpi creole side) and states that after aggressive normalization no eval sentence remains in training files.
Single-factor ablations with effect sizes
New human-original test coverage
Metric-validity finding beyond the leaderboard
Weaknesses
Silver-reference circularity for two of three languages
Reproducibility gaps
Sparse uncertainty on main results
Narrow baseline breadth
FLORES deficit explanation is speculative
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a shared-task system paper: the University of Melbourne's WMT26 CreoleMT entry for Tok Pisin (tpi), Bislama (bis) and Solomon Pijin (pis), both directions. The central claims are (1) a two-stage fine-tune of NLLB-3.3B — broad pre-training on a decontaminated 596k-pair religion-heavy union, then continued fine-tuning on a domain-balanced mix with spoken transcripts and 6k Gemini-distilled pairs — beats every open baseline by 3+ chrF++ in all six direction-by-test-set cells with human-original references; (2) it matches Gemini 3.5 Flash on Bouquet tpi (51.5 vs 51.9) and leads on spoken transcripts (47.7 vs 41.0), trailing only on FLORES (47.6 vs 50.2); (3) distillation gains saturate within about 1.5k pairs; and (4) three generalizable findings: chrF++ over-weights orthography, domain coverage transfers across related creoles, and joint training helps the lowest-resource languages.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What languages does this WMT26 system translate?
Tok Pisin (Papua New Guinea), Bislama (Vanuatu) and Solomon Pijin (Solomon Islands), in both English-to-creole and creole-to-English directions, using one joint NLLB-3.3B-based model per direction that covers all three languages.
How much better is it than existing translation tools?
On human-original test sets (Bouquet Tok Pisin and 300-row spoken transcripts per language) it beats open baselines such as NLLB-3.3B, Kreyòl-MT and CreoleM2M by at least 3 chrF++ in every direction, roughly matches Gemini 3.5 Flash (51.5 vs 51.9 on Bouquet; 47.7 vs 41.0 on transcripts), and trails Gemini on FLORES (47.6 vs 50.2).
Are the evaluation sets trustworthy?
Partly. Bouquet Tok Pisin uses human references and the spoken transcripts are human-original, but Bouquet references for Bislama and Solomon Pijin were generated by Gemini — the authors themselves call these silver sets unsatisfactory, so those scores should be read with caution.
What was the single biggest quality improvement in the paper?
ResPELLing Solomon Pijin loanwords into the local spelling convention used by the references: +7.7 chrF++ from a spelling-only change, which is also why the authors question whether chrF++ measures real translation quality or mere orthographic conformity.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's curve values are known only from the §5.1 prose (+1.54, +2.73, +4.3 ESA); the figure itself extracted as axis labels only.
- The ‡ marker on 'Gemini 3.5 Flash' in Tables 2–3 has no corresponding footnote text in the extraction (the caption explains only the * submitted-system row).
- An arXiv stamp is interleaved mid-sentence ('generalise be-arXiv:2609.13615v1...yond this task'); cosmetic and did not affect reading. The text otherwise runs complete through the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.