Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Following
Niklas Scholz, David Thulke, Abdallah Nasir, Will Allred, Evgeny Matusov, Hermann Ney
cs.CL · cs.LG
Quick verdict
Compares 11 forgetting-mitigation methods for MT fine-tuning; EWC preserves general benchmarks but not MT instruction controls. Solid single-run evidence.
Summary
The paper asks whether catastrophic-forgetting mitigation methods validated on general benchmarks still work when the fine-tuning task is machine translation, and whether they preserve MT-specific instruction following (MT-IF): formality, grammatical-gender, and length control. Eleven methods — grouped by anchor into auxiliary-data (data mixing, merging, EWC/EWC-DR), model-output (KL, STM, confident-conflict masking, EAFT), and parameter-anchor (freeze, SPD, LoRA) — are screened on Llama 3.2 1B Instruct across three unidirectional language pairs, then the strongest configurations are applied to Llama 3.1 8B Instruct on bidirectional Arabic-English and Spanish-English data (1M pairs). On the 8B Spanish model, EWC limits the general-benchmark drop to 1.7 points versus 11.0 for standard SFT, yet its MT-IF average (35.3) stays near SFT (32.3), far below the base model (71.1). Only data mixing with control-task examples restores a control (e.g., gender accuracy 77.9% vs 17.8% after SFT), but gains vanish with a different prompt (formality 53.8 vs 95.5) and do not transfer across tasks; SPD is a partial exception. The Fisher analysis (Figure 4) supports a gradient-based explanation. Limitations: single runs without variance estimates, confounded Stage 1-to-2 design changes, lexical-matching MT-IF metrics, and no released code.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Retention on general benchmarks does not establish that MT-specific instruction following is preserved after MT fine-tuning.
- EWC best preserves general capabilities on both model sizes (8B Spanish: -1.7 vs -11.0 for standard SFT) but leaves MT-IF near the SFT baseline, even when Fisher estimation includes control-task examples.
- Only data mixing with control-task examples preserves the corresponding control, and the effect is tied to seen prompts and does not transfer across control tasks.
- Methods anchored to model outputs fail because their constraints apply only on MT training inputs, leaving evaluated behavior unconstrained.
Strengths
Novel and practically important evaluation target
Section 6.3.2 demonstrates a dissociation no prior mitigation evaluation measured: EWC retains general capabilities (65.1 vs SFT 55.8 on 8B Spanish) while its MT-IF average (35.3) stays within 3 points of SFT (32.3) and 36 points below the base model (71.1).
Broad, structurally fair comparison
Eleven methods from 2017-2026 literature unified under an anchor taxonomy (Section 3), all evaluated under an identical SFT recipe (loss masked to assistant turns, Appendix A.2); Stage 1 alone spans 95 configuration runs across three language pairs (Figure 2).
Self-critical prompt-memorization ablation
Mechanistic probe with a control condition
Unusually transparent reporting
Weaknesses
No variance quantification anywhere
Confounded Stage 1-to-2 transition
MT-IF metric validity limits
Narrow MT-IF coverage
No code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies a narrow, well-posed question: do catastrophic-forgetting mitigation methods, validated in the literature by retention on general benchmarks, transfer when the fine-tuning task is machine translation — and do they preserve MT-specific instruction following (MT-IF): formality, grammatical-gender, and length control applied to translations (Sections 1, 4.3)? Eleven methods are organized by a unifying anchor taxonomy (Section 3): auxiliary data (Data Mixing, Model Merging, EWC/EWC-DR), model outputs (KL regularization, Selective Token Masking, Confident Conflict masking, EAFT), and base parameters (Freeze, SPD, LoRA). The central claim — supported by the Stage 2 data — is that general-benchmark retention and MT-IF retention dissociate: EWC cuts the 8B Spanish general-benchmark drop to 1.7 points (vs. 11.0 for standard SFT) while its MT-IF average (35.3) sits within 3 points of SFT (32.3) and 36 points below the base model (71.1). This is a genuine, practically useful finding.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MT-specific instruction following (MT-IF)?
Tasks where a natural-language instruction modifies a translation, such as producing a formal or informal variant, choosing the grammatical gender forms for a person mentioned in the source, or making the translation shorter or longer than the source. The paper tests these via CoCoA-MT formality, MT-GenEval gender, and FLORES with length instructions.
Does EWC prevent catastrophic forgetting when fine-tuning LLMs for translation?
For general capabilities, largely yes: on Llama 3.1 8B fine-tuned on Spanish-English, EWC's average drop over eight general benchmarks was 1.7 points versus 11.0 for standard fine-tuning. But the paper shows EWC does not protect translation-specific controls (formality, gender, length): those stay near standard-fine-tuning levels even when the Fisher importance data includes control-task examples.
How can I preserve formality or gender control when fine-tuning a translation LLM?
According to this paper, the only effective approach among the eleven tested is to mix examples of that specific control task into the fine-tuning data — and even then the skill is largely tied to the exact prompt format seen during training (a different prompt dropped formality accuracy from 95.5 to 53.8) and does not extend to other controls.
Is code or data available for this paper?
The paper documents all training prompts (Appendices A.4-A.7), per-corpus data compositions, hyperparameters, and compute budgets, and the main Stage 2 experiments use public data (OPUS, Tulu 3, CoCoA-MT, MT-GenEval). The extracted text does not mention a released code repository, and Stage 1 uses an internal data-preparation pipeline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 could only be assessed from embedded numbers and captions; plot-level detail (Pareto shapes, marker labels) may be partially lost in extraction.
- The Table 18 Merge row duplication may be an extraction artifact rather than a paper error, though identical COMET and BLEU across both columns makes a source-side duplication more likely.
- Several cited works are 2026 venues (ICML/ACL/ICLR 2026) that could not be verified from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.