Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation
cs.CL · cs.AI
Quick verdict
Two-stage MQM-guided MT post-edit framework with solid ablations, but its 'consistent gains' claim conflicts with its appendix; human evaluation unreported.
Summary
This Amazon industry paper proposes a two-stage framework for domain-specific machine translation post-editing that decouples diagnosis from repair: a retrieval-augmented LLM evaluator (Claude Opus 4.5) outputs structured, span-level MQM diagnoses with suggested fixes under an explicit 'edit contract', and a separate post-editor LLM applies only those minimal edits. Experiments run on an internal e-commerce translation memory across six EN→X directions (5,000 test segments each), with seven post-editor LLMs from three providers; the best configuration (Claude Sonnet 4.5) improves COMET-22 in all six directions (e.g., +3.54 EN→DE, +7.92 EN→ES) and beats a one-stage judge-and-refine baseline on EN→DE and EN→CN. Ablations attribute the largest gains to the RAG-augmented prompt, show monotonic benefits from stronger evaluators, and—usefully—show that a second post-edit round degrades quality in 6 of 7 configurations. The main concerns: the abstract's claim of consistent improvement on both metrics is contradicted by Appendix A (COMETKiwi drops for EN→IT and EN→JP; one-stage wins COMET-22 on EN→FR/IT/JP); the human-evaluation agreement results promised in the abstract appear nowhere in the text; no significance tests are reported; and proprietary data with no code release makes reproduction infeasible.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Decoupling MQM evaluation from post-editing via an explicit edit contract yields larger and more controllable quality gains than one-stage judge-and-refine post-editing.
- The best two-stage configuration improves COMET-22 over the no-post-edit baseline in all six EN→X directions and outperforms one-stage post-editing on EN→DE and EN→CN.
- The retrieval-augmented evaluator prompt (P3, with rubric and edit contract) is the dominant source of improvement over plain GEMBA-MQM.
- A single evaluator-guided post-edit round suffices; a second round degrades COMET-22 in 6 of 7 editor configurations on EN→DE.
- The evaluator's error spans and severities agree strongly with human MQM annotations and human editor preferences (claimed in the abstract; not reported in the provided text).
Strengths
Clean architectural separation with an explicit edit contract
Sec 3.2 and Appendix E specify how evaluator diagnoses bind the editor's behavior; the B.2 control (one-stage GEMBA-MQM: +0.12 COMET-22 on EN→DE vs +3.54 for the two-stage pipeline) directly supports the core design choice.
Systematic multi-provider model sweep
Seven editors across Google/Anthropic/OpenAI (Tables 2 and 5) yield practical guidance: no single winner (Gemma 3 4B is best on EN→CN despite worst on EN→DE), with Claude Sonnet 4.5 strongest overall.
Honest negative result on iteration
Evaluator ablation with cost implications
Above-average transparency for an industry paper
Weaknesses
Headline consistency claim contradicted by the paper's own appendix
Unsubstantiated human evaluation
No statistical inference
Compute/capacity confound in the central comparison
Low external reproducibility and selective presentation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Amazon industry paper (arXiv 2609.22793) proposes a two-stage post-editing pipeline for domain-specific MT: a retrieval-augmented LLM evaluator (Claude Opus 4.5) produces structured, span-level MQM diagnoses with suggested fixes under an explicit "edit contract" (Sec 3.2, Appendix E), and a separate post-editor LLM applies minimal repairs restricted to those diagnoses. Four claims are made: (1) the two-stage design beats one-stage judge-and-refine; (2) the best configuration "consistently improves both COMET-22 and COMETKiwi over one-stage methods"; (3) evaluator spans and severities show strong agreement with human MQM annotations and human editor preferences; (4) one post-edit round suffices.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a two-stage MQM-guided post-editing framework for machine translation?
A pipeline where one LLM first evaluates a machine translation and outputs structured MQM-style error diagnoses (category, severity, span, suggested fix) constrained by an 'edit contract', and a second LLM then applies only those minimal targeted fixes — instead of asking a single model to judge and rewrite in one step.
Did the framework actually improve translation quality in the paper's experiments?
On Amazon's internal e-commerce data (six EN→X directions, 5,000 test segments each), the best configuration raised COMET-22 in all six directions (e.g., +3.54 for EN→DE, +7.92 for EN→ES) and beat the one-stage baseline on EN→DE and EN→CN. But COMETKiwi fell for EN→IT (-1.65) and EN→JP (-0.99), and the one-stage approach scored higher COMET-22 on EN→FR, EN→IT and EN→JP, so the gains are not uniform.
How many rounds of LLM post-editing are needed?
According to the paper's Tables 3-4, one round is enough: editors flagged only 5-32% of first-round outputs as still erroneous, and a second round lowered COMET-22 in 6 of 7 configurations on EN→DE (up to -1.86), with only one configuration gaining (+0.14).
Can this framework be reproduced?
Not exactly. It was evaluated on proprietary Amazon e-commerce translation memories with no code release, so the paper's numbers cannot be re-derived externally. However, the full prompts and Bedrock model IDs are published in the appendices, so practitioners can reimplement the method on their own data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extraction contains no figures; all quantitative evidence comes from text-rendered tables (Tables 1-10), which appear complete through Appendix E.
- No section reporting the human MQM agreement or human editor preference study promised in the abstract appears in the extracted text (Sections 1-6 plus Appendices A-E); if such a section was lost in extraction, the corresponding weakness and risk flag should be re-weighed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.