StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translation
Rohit Dhaipule, Sukhdeep Singh Kharbanda, Prasanth Bathala, Pradyumna Lanka, Anubhav Shrimal
cs.CL · cs.LG
Quick verdict
StalePO adds downward bias, anchoring and token KL to DPO for stale MT post-edits; +14.9pp En-Hi gain is promising but single-seed, judge-scored, private data.
Summary
The paper identifies the 'stale preference' problem in industrial machine translation: the only available human preference data consists of post-edits made on outputs of older, weaker NMT systems, so the 'preferred' response is often inferior overall to the current model's output while remaining better on a few dimensions (e.g., style-guide adherence). The authors argue standard DPO fails here for three reasons: it does not constrain the direction of likelihood change, offers no anchor to the model's own better generations, and aggregates KL at sequence level. StalePO combines three remedies in one objective: an APO-Down-style term biasing likelihood downward on both pair members, a ReLU-gated anchor on a sampled base response yb, and TDPO1-style token-level sequential forward KL. On English→Hindi localization data (24k filtered pairs, GPT-OSS 120B with QLoRA), StalePO raises the LLM-as-judge MQM all-pass rate by 14.9 pp (Style +20.9, Fluency +16.5, Accuracy −5.4); on unfiltered English→Turkish data the gain is 4.6 pp. A human MQM evaluation on a 2,000-segment En→Hi subset confirms +13.8 pp. Ablations show each component alone yields changes indistinguishable from the base model (DPO −0.4, TDPO +0.5, BAPO −0.2, APO-Down +0.6 pp), while BAPO-Down reaches +7.4 pp. Limitations: single training run, no significance tests, no human validation for En→Tr, proprietary data, undisclosed absolute pass rates.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Preference data inherited from weaker predecessor MT systems ('stale preferences') is systematically inferior overall to the current model's output yet retains signal on a few quality dimensions.
- Standard DPO structurally fails in this regime because it lacks directional control, base anchoring, and token-level KL constraints.
- StalePO's three mechanisms are jointly necessary: each alone leaves performance indistinguishable from the base model, while the combination converts stale feedback into gains.
- StalePO improves LAJ-MQM all-pass rate by +14.9 pp on En→Hi and +4.6 pp on En→Tr, with human MQM confirming +13.8 pp on a 2,000-segment En→Hi subset.
- Gains concentrate on style and fluency with a modest, disclosed accuracy regression, consistent with the stale-preference formulation.
Strengths
Real industrial problem with formal grounding
Sec. 2 defines stale data via a quality-dimension subset S; Appendix A empirically quantifies the premise (post-edits closer to legacy NMT: mean TER 0.426 vs 0.517 to the new base model, paired t=46.9, p≪0.001; 55.3% vs 23.7% segment-level proximity).
Human evaluation corroborates the LLM judge
A 2,000-segment (24,446 words) En→Hi human MQM study agrees with LAJ-MQM in direction and concentration: all-pass +13.8 pp, Style +15.9, Fluency +37.3, Accuracy −3.5 (Table 1 bottom block, Sec. 5.1).
Component-isolated ablation design
Honest reporting of trade-offs and metric inversion
Bias-aware evaluation setup
Weaknesses
No statistical inference behind headline deltas
Thin En→Turkish evidence
All-pass metric can mask a real quality loss
Low reproducibility
Coarse sensitivity analysis on one pair only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
StalePO: Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the stale preference problem in machine translation?
When an MT system is upgraded, the only human preference data available is post-edits made on outputs of the older, weaker system. Those 'preferred' translations are often worse overall than what the new model already produces, but still better on a few dimensions such as style-guide adherence. The paper calls this the Stale Preference problem and argues standard DPO cannot learn from such data.
How much does StalePO improve translation quality?
Using an LLM-as-judge MQM evaluation, StalePO raised the fraction of segments passing all seven quality checks by 14.9 percentage points on English→Hindi and 4.6 points on English→Turkish versus the base model. A human evaluation on a 2,000-segment English→Hindi subset confirmed a 13.8-point gain, concentrated in style (+15.9) and fluency (+37.3), with a small accuracy drop (−3.5).
How is StalePO different from DPO or APO-Down?
StalePO adds to DPO-style training: (1) an APO-Down-style term biasing likelihood downward on both responses, (2) a ReLU-gated anchor keeping the model close to its own base response, and (3) TDPO1-style token-level sequential KL. In the paper's ablations each piece alone yields almost no change (DPO −0.4, APO-Down +0.6 pp), the anchor is the biggest contributor (BAPO-Down +7.4), and the full combination reaches +14.9 pp on English→Hindi.
What are the limitations of StalePO?
It only helps when stale post-edits still beat the model on some dimensions (an empty advantage set gives no signal); it is unsuited to the normal case where post-edits are uniformly better than model output; the English→Turkish results lack human validation; training uses proprietary Amazon data with undisclosed absolute pass rates; and there is a measured accuracy regression (−5.4 pp under the LLM judge) traded against style and fluency gains.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are available only through captions and prose; underlying curve values (e.g., exact sensitivity magnitudes in Fig. 2) could not be verified from the extracted text.
- Extraction appears complete through Appendix A and references, but page-level truncation cannot be fully ruled out.
- All results are deltas relative to the base model; absolute pass rates are withheld by company policy, so numbers cannot be benchmarked against other papers.
- The human MQM study details (annotator count, agreement) are absent from the text, so its reliability is assessed only from reported deltas.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.