From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models
Shuo Cai, Yanggan Gu, Zihao Wang, Yuanyi Wang, Yibo Yan, Wenjun Wang, Yuhang Liu, Guanghao Zhu, et al.
cs.CL
Quick verdict
Claims first unified 3-level taxonomy of LLM model fusion; clear definition and honest comparisons, but taxonomy boundaries and selection method are debatable.
Summary
This survey proposes 'model fusion' as an umbrella term for methods that build a single target model from multiple source models, defined operationally by three admissible operations — parameter combination, representation alignment, and behavior distillation — with the requirement that the fused model runs at inference without the complete sources. Under this definition, model merging becomes parameter-level fusion and knowledge distillation becomes behavior-level fusion, while ensembles, routing, LLM-as-a-judge, and reward-only RL are excluded. The paper reviews over 150 papers, organizes methods into the three levels (Sections 3.1-3.3), and formalizes each level's fusion mapping (Equations 4-6). It reviews metrics (average vs. normalized performance) and 14 benchmark resources (Table 8), noting that reusable evaluation protocols concentrate in parameter-level settings. Practical takeaways (Section 4) deliberately restrict comparisons to within-paper evidence (Tables 2-3), showing e.g. parameter-level TIES at 61.00 nearly matches behavior-level MT-OPD at 60.46 despite 967.9 GPU-hours, and that hybrid pipelines like FeatCal raise Task Arithmetic from 63.5 to 65.8. Challenges cover missing applicability theory, heterogeneous alignment, per-capability retention evaluation, and risk transfer (backdoors, PII leakage). Limitations are candid: no controlled head-to-head numbers, no variance in many sources, boundary cases admit alternative labels. Main weaknesses: no systematic search methodology, contested taxonomy placement of classic KD methods, and noticeable self-citation of the authors' own methods.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Model fusion admits a unified operational definition — parameter combination, representation alignment, or behavior distillation, plus inference independence — that subsumes model merging and knowledge distillation while excluding ensembles, routing, LLM-as-a-judge and reward-only RL (Section 2).
- Existing surveys cover at most two of the three fusion levels; this is the first to treat parameter-, representation-, and behavior-level fusion together (Table 1).
- No fusion level dominates: cross-level rankings flip with evaluation setting and compute budget, so only within-paper comparisons are meaningful (Section 4, Tables 2-3).
- Adding a second fusion level to a pipeline improves reported scores in all four documented cases (Table 3: 63.5→65.8, 58.74→60.34, 87.0→87.5, 73.5→75.6), but gains are setting-dependent.
- Reusable evaluation protocols remain concentrated in parameter-level fusion; future benchmarks should report per-capability retention, worst-task degradation, and fusion cost (Sections 3.4, 5).
Strengths
Operational definition with explicit, falsifiable exclusions
Section 2 admits only three construction operations plus an inference-independence condition; ensembles, RouteLLM-style routing, LLM-as-a-judge and reward-only RL are explicitly excluded, and the on-policy-distillation boundary (dense token-level teacher signal required) is the sharpest clause in the paper.
Disciplined comparison policy
Section 4 (Takeaways 2-3) restricts all cross-level claims to within-paper comparisons (Tables 2-3) and labels cross-paper numbers 'illustrative rather than controlled'; this avoids the genre's usual league-table fallacy.
Structured reference apparatus
Well-supported evaluation-gap finding
Current and industry-anchored
Weaknesses
No systematic search methodology
Taxonomy boundary tension
Thin evidential base for takeaways
Formalism without predictive power
Self-citation concentration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines model fusion operationally (Section 2): a target model is built from sources via parameter combination, representation alignment, or behavior distillation, and must run at inference without the complete sources (p_tgt(y|x,S)=p_tgt(y|x)). This single boundary does real work: it admits model merging as parameter-level and knowledge distillation as behavior-level fusion, while excluding ensembles, RouteLLM-style routing, LLM-as-a-judge, and reward-only RL; the on-policy-distillation clause (dense token-level teacher signal required) is the sharpest and most useful line drawn. Table 1 positions the survey against ten predecessors (Gou et al. 2021 through Fang et al. 2026), claiming to be the only one covering all three levels. The contributions are organizational: the definition (Eqs. 1-6), the taxonomy (Section 3, Figure 3), evaluation and benchmark reviews (Section 3.4, Tables 8-9), four takeaways (Section 4, Tables 2-3), and four challenge areas (Section 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between model fusion and model merging?
In this survey's terminology, model merging is one subtype of model fusion. Model fusion is the umbrella term for building one target model from multiple source models via three possible operations: combining parameters (parameter-level, e.g., merging weights), aligning intermediate representations (representation-level, e.g., repairing hidden-state drift), or distilling output behaviors (behavior-level, e.g., knowledge distillation). The target must run at inference without the complete source models.
Do model ensembles or LLM routing count as model fusion?
No. Under the paper's operational definition, both ensembles (averaging predictions from multiple models) and routing systems like RouteLLM (selecting a model per query) keep and invoke the source models at inference time, so they fail the requirement that the fused target operate independently of the complete sources. LLM-as-a-judge is also excluded because it evaluates outputs rather than constructing a target model.
Is there code or a dataset released with this survey?
The paper releases a curated, continuously updated list of model-fusion papers on GitHub (Awesome-Model-Fusion-Survey) rather than code. The benchmarks it reviews, such as FusionBench and MergeBench, are separate open resources with their own code repositories.
Which fusion level should I use for my models?
Per the paper's practical takeaways: if source models share architecture, initialization and tokenizer and weights are available, parameter-level merging is the cheapest first option; if hidden-state drift appears, add representation-level repair; if only outputs are available or models differ in architecture or tokenizer, use behavior-level distillation. No level dominates across settings, and combining levels helped in all four documented cases.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 (overview diagram, progress timeline with annual counts) are captioned but their graphical content is not analyzable from the extracted text.
- Table 2's CLIP/SAMerging block is garbled in extraction; the five-of-six-configurations reading was interpreted cautiously.
- Appendix D/E narrative text is minimal in the extraction; Tables 6-7 carry that content.
- Minor risk that some appendix prose (e.g., Appendix H detail) was truncated, though Appendix H appears substantively present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.