When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents
Syed Shariyar Murtaza, Yifan Nie, Utkarsh Soni, Eugene Wen, Arvid Frydenlund
cs.IR · cs.AI · cs.LG
Quick verdict
Study: synthetic-data LoRA causes forgetting in LLM-agent skill retrieval; regularized tuning preserves OOD, but low rank outweighs the regularizers.
Summary
This industry-track paper from Manulife studies whether fine-tuning compact retrieval models on synthetic supervision degrades skill retrieval for LLM agents. The authors build a production skill router over 34,396 skills and harvest real supervision by executing roughly 1,400 trials of 173 tasks from SkillsBench and Terminal-Bench 2.0 in Harbor, yielding 273 positive task-skill pairs across 75 tasks. Two synthetic tracks support training: Track A (1,669 anchor-skill synthetic tasks) and Track B (13,271 multi-positive rows generated by Claude Opus 4.7 and judged by GPT-5.5). Experiments on Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B show aggressive LoRA fine-tuning overfits the synthetic distribution while OOD recall collapses (0.850 to 0.650 on a 10-task Terminal-Bench ring). Four continual-learning regularizers — embedding anchors, LwF, EWC, and L2-init — preserve OOD performance (TOST-confirmed equivalence) and lift synthetic-ring Recall@10 by 13.98% relative. BEIR evaluation on six datasets and paired bootstrap, Wilcoxon, and sign-test analyses support the pattern. Main limitations: real evaluation rings are tiny (n=21, n=10); the lambda sweep shows the regularizers themselves add little over conservative low-rank LoRA; and judge agreement is weak (kappa=0.28). Code is released on GitHub, though the synthetic data depends on closed-weight APIs.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Synthetic-data LoRA fine-tuning of skill retrievers improves in-distribution retrieval but induces catastrophic forgetting on real/OOD tasks, e.g., Ring 3 Recall@10 falls from 0.850 (frozen) to 0.650 under aggressive r=16 LoRA (Table 2).
- Four continual-learning-inspired regularizers (embedding anchor, LwF, EWC, L2-init) under conservative LoRA preserve OOD performance — TOST-equivalent to the frozen 0.6B baseline — while improving synthetic-ring Recall@10 by 13.98% relative (0.5337 to 0.6083).
- LoRA rank is the primary factor controlling the performance-retention trade-off, whereas regularizer weight lambda has only a minor effect (Appendix C).
- A compact Qwen3-Embedding-0.6B matches a much larger 4B+BM25+RRF hybrid on the 75-task real pool (both HIT@10=0.907, RECALL@10=0.604, Table 1).
Strengths
Production-scale real supervision pipeline
A 34,396-skill catalog with ~1,423 Harbor-executed verifier-scored trials over 173 tasks (§2.2), distilled into a 75-task / 273-strict-positive-pair benchmark, with threshold sensitivity checked over [0.4, 0.7] — rare grounding for a retrieval fine-tuning study.
Unusual transparency about synthetic-data quality
Appendix B reports soft-fail quality gates in full: G7 shows synthetic vs real reward KS D=0.31 (p<10^-80), G8 shows only 0.52% of the catalog appears as positives, G9 reports inter-judge Cohen's kappa=0.28 — the paper bounds its own claims instead of hiding the mismatches.
Statistical rigor unusual for industry work
Consistent replication across regimes
Actionable deployment recipe with code
Weaknesses
Tiny real evaluation rings carry the headline claim
The four regularizers are nearly indistinguishable and possibly unnecessary
The 13.98% headline is relative and synthetic-only
Label and benchmark-selection fragility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This EMNLP 2026 Industry Track paper (arXiv:2609.10750, Manulife) asks whether fine-tuning compact retrieval models on synthetic supervision degrades skill retrieval for LLM agents. Claimed contributions: (A) a systematic study at production scale — 34,396 skills, 109 verifier-scored tasks; (B) a unified benchmark combining real and synthetic supervision; (C) an evaluation of four continual-learning-inspired mitigation strategies under a shared recipe. Headline claims: aggressive synthetic-data LoRA induces catastrophic forgetting (Ring 3 Recall@10 drops 0.850 → 0.650); regularized conservative fine-tuning preserves OOD behavior while lifting synthetic-ring recall by 13.98% relative; and LoRA rank, not regularizer weight, is the primary trade-off knob.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Fine-tuning LLM-agent skill retrievers on synthetic data improves retrieval on that synthetic distribution but causes catastrophic forgetting on real and out-of-distribution tasks — recall dropped from 0.850 to 0.650 in the most aggressive configuration. Conservative LoRA fine-tuning combined with continual-learning regularizers (embedding anchors, LwF, EWC, L2-init) preserves out-of-distribution performance while retaining most synthetic gains.
Are the regularizers actually better than just using a small LoRA rank?
Not clearly. The appendix shows that conservative LoRA with the regularizer weight set to zero achieves nearly identical results to all four regularized variants, and the authors state that LoRA rank — not the regularizer weight — is the main factor controlling the forgetting trade-off. The practical takeaway is 'low-rank, low-learning-rate, conservative LoRA'; the specific regularizers add little measurable benefit in these experiments.
Is the code and data available?
The authors provide a GitHub repository (manulife-ai/emnlp2026). However, the synthetic training tracks depend on closed-weight commercial LLMs (GPT-5, Claude Opus 4.7, GPT-5.5, Claude Opus 4.8), which the authors themselves note makes exact reproduction of the synthetic data sensitive to API drift.
What does the 13.98% improvement refer to?
It is a relative improvement in Recall@10 on the synthetic in-distribution evaluation ring — from 0.5337 with the frozen Qwen3-0.6B model to about 0.608 with regularized fine-tuning. Performance on real, out-of-distribution tasks is preserved (statistically equivalent to the frozen baseline), not improved.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction interleaves table headers and cell values (Tables 2, 3, 6, 7); all cited numbers were cross-checked against surrounding text but the visual table layout is reconstructed.
- arXiv listing metadata (categories cs.IR/cs.AI/cs.LG) was taken from the task prompt, not the extracted text, which shows only the cs.IR stamp.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.