LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Edits
Kathy H\"ammerl, Gabriel Bretschner, Joern Wuebker
cs.CL
Quick verdict
Adapts COMETKiwi to localisation QE via multi-task post-edit training and a tokeniser fix; good analysis and new datasets, but natural-data gains remain modest.
Summary
The paper addresses the drop in quality-estimation (QE) performance when reference-free metrics like COMETKiwi22 DA are applied to commercial localisation content instead of news/wiki text. The authors first document the problem: baseline COMETKiwi ranks a preferred post-edit above the machine-translation output only 53.1% of the time on internal localisation data (vs 66.2% on WMT24), is blind to non-breaking spaces its tokeniser strips, and barely penalises mistranslated long numbers. They adapt the model without human direct-assessment scores by converting post-edits into two signals: preference pairs (MT output vs human edit, margin ranking loss) and artificial continuous scores (chrF++ against the post-edit plus heuristics, MSE loss). A multi-task setup mixing both losses at a 2:1 MSE:MRL ratio mitigates each loss's failure mode: pure MRL collapses the score distribution and destroys global correlation, pure MSE hurts segment-level ranking. A tokeniser intervention preserving spaces and adding NBSP/ZWSP/ZWNJ tokens yields 4-10 point gains on their LocCheck challenge set. They release LocCheck and LocHD (ESA annotations, 300 segments x 10 languages). With 5k augmented minimal pairs, LocCheck accuracy reaches ~90%, but natural-data preference accuracy drops (59.6% to 57.4%). Limitations: single base model, internal training data unpublished, no significance testing or multi-seed variance, and LocHD correlation still below a GEMBA-ESA LLM baseline.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- COMETKiwi22 DA suffers severe domain shift on localisation data: preference accuracy drops from 66.2% (WMT24) to 53.1% (internal data), and the model is blind to non-breaking spaces, whitespace preservation, and long-number errors.
- Multi-task fine-tuning combining MSE regression on chrF++-derived artificial scores with MRL on post-edit preference pairs (2:1 ratio) outperforms either loss alone because the two signals stabilise absolute scores and within-source ranking respectively.
- A targeted tokeniser intervention (preserving spaces, adding NBSP/ZWSP/ZWNJ and directional-marker tokens) is necessary for the model to perceive localisation surface issues, adding 4-10 points on LocCheck.
- Data augmentation with minimal pairs is required for long-tail localisation rules, raising LocCheck accuracy to roughly 90% on both splits, at a small cost to natural-data preference accuracy.
- Two new resources, the LocCheck challenge set and the LocHD ESA-annotated corpus across ten target languages, enable evaluation of localisation-specific QE behaviour.
Strengths
Precise diagnosis of QE failure modes in localisation
Table 1 gives concrete blind spots: identical COMETKiwi scores (84.83) for a French sentence with and without a non-breaking space, a higher score for a hallucinated cloze fill (84.20 vs 78.89), and near-identical scores for 7777777 vs 777777 digits. Figure 1/3 quantify the domain gap: 66.2% preference accuracy on WMT24 vs 53.1% on internal localisation data.
Insightful analysis of loss interaction
Section 5.2 and Figure 5 explain why MRL-only training compresses scores into a band roughly twice the margin, collapsing global Kendall on LocHD to 0.0442 and ACES to 3.67 (Table 6), and why multi-task MSE+MRL restores score spread while improving positive/negative separation - a practically useful finding for anyone adding ranking losses to regression metrics.
Simple, targeted tokeniser intervention with clear effect
New public resources plus a public-data replication
Thorough ablations with stated decision rationale
Weaknesses
Challenge-set circularity in the headline gains
No statistical treatment of results
Adapted model still loses to an LLM baseline on human-score correlation
Noisy preference signal only partially validated
Single base model and modest natural-data gains vs abstract framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LocQE and how does it differ from COMETKiwi?
LocQE is COMETKiwi22 DA fine-tuned for localisation use cases. It adds a modified tokeniser that preserves leading/trailing/multiple spaces and can represent special characters like non-breaking spaces, and it is further trained on roughly 10-20k post-edit examples using two losses at once: a ranking loss on preference pairs (machine translation output vs the human post-edit) and a regression loss on chrF++-derived scores. The result ranks preferred translations far more reliably on localisation content and can detect formatting and number issues the original model cannot see.
What new datasets does the paper introduce?
Two. LocCheck is a challenge set of minimal pairs (about 2,564 internal and 841 public items) covering localisation-specific issues such as long-number mistranslation, leading/trailing spaces, French non-breaking spaces, all-caps handling, URL and do-not-translate copying, across 11 target languages; code to regenerate the public split is released. LocHD provides 3,000 error-span-annotation (ESA) quality scores from professional translators for 300 English-source segments translated into ten languages.
Why do quality estimation models fail on localisation content?
The paper gives two reasons. First, domain shift: models trained on news/wiki direct assessments rank same-source translations barely above chance on localisation post-edit data (53.1% vs 66.2% on WMT24). Second, architectural blindness: the default XLM-R tokeniser strips leading/trailing/multiple spaces and normalises non-breaking spaces, so formatting errors are literally invisible to the model, and long digit sequences are handled in ways that make mistranslations hard to distinguish.
Do preference-based and score-based fine-tuning signals conflict when training translation metrics?
Yes, when used alone. Training only on preference pairs makes the model compress its output scores into a narrow band roughly the size of the ranking margin, which destroys global (across-source) score correlation; training only on regression scores weakens same-source ranking. The paper shows that mixing both signals in multi-task training makes them stabilise each other, with a 2:1 ratio of regression to ranking data performing best.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were available only as extracted text labels; the scatter plots (Figure 5) and Pareto curve (Figure 6) could not be visually inspected.
- The internal training data and LocCheck internal split cannot be independently examined; assessment of their quality relies on the paper's descriptions and the small Appendix A.4 validation.
- Minor extraction artefacts exist (e.g., Table 11 header layout), but key numbers cross-check consistently between Figure 1/3 and Tables 3, 6, and 12.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.