Can One Adapted Model Do It All? Fine-Tuning Strategy Selection for Customer Support LLMs
Md Tahmid Rahman Laskar, Xue-Yong Fu, Xue-Yong Fu, Shashi Bhushan TN, Shashi Bhushan TN
cs.CL
Quick verdict
Controlled comparison of fine-tuning strategies across 13 support LLMs finds MT-full best; useful, but lacks significance tests and half the data is private.
Summary
The paper presents a controlled comparison of adaptation strategies for customer-support LLMs: single-task (ST) and multi-task (MT) full fine-tuning, LoRA variants of both, sequential fine-tuning, and post-hoc model merging. Thirteen instruction-tuned models from five families (Qwen3, Qwen3.5, Gemma-3, Llama-3.1, Mistral), 0.6B-32B parameters, were fine-tuned on eight datasets — four public (Bitext intent classification, SAMSum, WixQA, When2Call) and four proprietary Dialpad tasks — totaling ~74.5k training and 8.7k evaluation samples and over 200 checkpoints. The central finding: MT full fine-tuning matches specialists on average (suite gaps within 2.8 points) while needing one checkpoint and no router; specialists collapse off-task (a Qwen3-8B chatbot-response specialist falls from 42.3 to 14.4 on summarization), so routing is essential. Sequential full fine-tuning forgets severely (intent classification 99.5 to 25.9) while sequential LoRA largely mitigates it (78.6). Merging a specialist with its base improves off-task robustness at a cost of at most 2.6 in-task points for 4B+ models, with alpha in [0.15, 0.30] recommended; multi-specialist TIES merging reaches only 41.2 versus 58.1 for MT-Full. Conclusions hold across ROUGE-1, BERTScore, and an LLM judge. Limitations: no significance testing or variance reporting despite eval sets as small as 200, full grid only for a seven-model core, English-only data, and an unverifiable proprietary internal suite.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Multi-task full fine-tuning is the strongest operational default at every tested model size, staying within 2.8 suite-average points of single-task specialists while needing one checkpoint and no router.
- Task specialists degrade sharply off-task (e.g., a Qwen3-8B chatbot-response specialist falls from 42.3 to 14.4 on summarization), so specialists are unsafe to deploy without reliable routing.
- Sequential full fine-tuning causes severe catastrophic forgetting (intent classification 99.5 to 25.9), while sequential LoRA preserves earlier skills far better (retains 78.6).
- Linearly merging a specialist with its base model improves off-task robustness with at most a 2.6-point in-task loss for models of 4B and above, with alpha in [0.15, 0.30] recommended.
- Merging multiple specialists via TIES (41.2 public-suite average) cannot substitute for multi-task fine-tuning (58.1).
Strengths
Unmatched breadth under one protocol
Thirteen models from five families, 0.6B-32B, 200+ checkpoints trained with identical hyperparameters and evaluated on one suite (Section 3, Tables 1 and 6) — prior strategy comparisons rarely control model family and size this way.
Deployment-grounded analysis
Cross-task matrices (Figure 2) show specialists collapse off-task (Qwen3-8B chatbot-response specialist: 42.3 in-task vs 14.4 on summarization) and that output-format similarity, not topic, predicts transfer; Section 4.6 and Appendix E translate findings into endpoint, routing, and training-cost guidance.
Actionable merging trade-off
Metric robustness and honest limitations
Weaknesses
No uncertainty quantification
Hyperparameter confound
Coverage gaps in the grid
Unexamined negative result
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an industrial empirical study (Dialpad Inc.) framed around a deployment question: for customer-support LLMs serving multiple skills (intent classification, knowledge-base QA, summarization, tool-use decisions), should teams train task specialists or one multi-task (MT) model? Under a fixed protocol the authors compare five adaptation strategies — single-task (ST) full fine-tuning, ST-LoRA, MT-Full, MT-LoRA, and sequential fine-tuning — plus two post-hoc merging schemes (linear merge-with-base; TIES for multiple specialists), across thirteen instruction-tuned models from five families (Qwen3, Qwen3.5, Gemma-3, Llama-3.1, Mistral; 0.6B-32B) and eight English datasets (four public: Bitext IC, SAMSum DS, WixQA QA, When2Call CS; four proprietary Dialpad tasks), totaling ~74.5k train / 8.7k eval samples and 200+ checkpoints. Headline claims: (1) MT-Full is the best operational default at every tested size; (2) specialists collapse off-task, so reliable routing is mandatory; (3) sequential full fine-tuning forgets severely while sequential LoRA mitigates; (4) merging a specialist with its base buys off-task robustness cheaply for models of 4B and above; (5) multi-specialist TIES merging cannot replace joint training.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Should I train one multi-task model or a separate specialist model per customer-support task?
According to this paper's experiments across 13 models (0.6B-32B), multi-task full fine-tuning matches per-task specialists on average within about 3 points while requiring only one checkpoint and no router. Specialists are strong on their own task but can collapse off-task (e.g., a chatbot-response specialist drops from 42.3 to 14.4 on summarization), so they should only be deployed with reliable request routing.
Does sequential fine-tuning cause catastrophic forgetting in LLMs?
Yes, for full fine-tuning: sequentially fine-tuning on four tasks in order dropped intent classification accuracy from 99.5 to 25.9 in this paper. Sequential LoRA retained 78.6, so LoRA-based updates are much safer, and the authors advise training the most critical task last (a reversed-order check confirmed earlier tasks suffer most).
What does merging a fine-tuned model with its base model do?
Linear interpolation theta_merged = alpha*theta_FT + (1-alpha)*theta_base improves off-task robustness. For models of 4B parameters and above, the in-task loss at alpha=0.5 is at most 2.6 points; a finer sweep suggests alpha in [0.15, 0.30] as the best robustness/specialization trade-off, since in-task benefit saturates by alpha~0.3 while off-task skills collapse beyond it.
Is the paper's data and code public?
Partly. The four public datasets (Bitext intent classification, SAMSum, WixQA, When2Call) are reproducible: the authors release training/inference configs, task prompts, preprocessing scripts, and training/evaluation code. The four internal Dialpad datasets are PII-redacted proprietary call transcripts and cannot be released, so internal results are supporting evidence rather than an independently verifiable benchmark.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 appear in the extraction as raw number lists; groupings were reconstructed from captions, so figure-derived values (e.g., Figure 4: 38.4/41.2/40.4 vs MT-Full 58.1) carry minor mis-assignment risk.
- The extraction appears complete from abstract through Appendix E; footnote URLs and exact table layouts may differ from the PDF.
- The promised code/config release is not included in the text; reproducibility claims rely on the authors' Reproducibility statement.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.