CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging
Mingjie Zheng, Zihao Chen, Wenqing Chen, Weile Yuan, Zhixuan Chu, Jianxing Yu, Zibin Zheng
cs.AI · cs.CL
Quick verdict
CoMerge learns 1,445 merging coefficients via DPO on self-made preference pairs; strong 0.9968 MergeBench NP but single-seed, gains partly fail to generalize.
Summary
CoMerge reformulates multi-task LLM merging as preference optimization. Instead of labels or entropy, it synthesizes preference pairs self-supervisedly: the chosen response comes from the relevant expert model, the rejected response from a naive Task Arithmetic merge (α=0.4) whose outputs exhibit interference-induced failure modes. DPO then optimizes tensor-wise merging coefficients — only 1,445 scalars for Llama-3.1-8B-Instruct with five experts — after task vectors are compressed by global top-10% magnitude pruning and selective SVD (rank ≤1500). On MergeBench (Llama-3.1-8B-Instruct), CoMerge reaches 0.9968 average normalized performance, above all evaluated merging baselines (Task Arithmetic 0.9538, AdaMerging Gen. 0.9788) and close to Full-DPO (0.9983), with the largest gains on instruction following (0.9952 vs 0.8687) and safety (1.0340). Ablations show both negatives and tensor-wise granularity matter; base-model negatives drop NP to 0.9497. Cost to peak is 38.3 GPU-minutes on one H20 versus 95.9 on four H20s for Full-DPO. On smaller backbones, gains persist on Llama-3.2-3B (0.9699±0.0032) but Gemma-2-2b-it (0.9321±0.0014) is only comparable to RegMean++. Key limitations: the headline 8B result is single-seed, no significance tests are reported, iterative-data gains are sweep-selected on test, and hyperparameter sensitivity is untested; code is promised at publication.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reformulating merging as preference optimization with Task-Arithmetic-derived hard negatives enables learning only 1,445 tensor-wise merging coefficients without external annotations or reward models.
- CoMerge achieves 0.9968 average normalized performance on MergeBench with Llama-3.1-8B-Instruct, above all evaluated merging baselines and near Full-DPO (0.9983).
- The gain is attributable to merging-specific negatives: replacing them with base-model outputs drops average NP from 0.9968 to 0.9497 (Tables 2 and 6).
- CoMerge reaches peak performance with 60.1% lower GPU-resource cost than Full-DPO (38.3 vs 95.9 GPU-minutes; 1 vs 4 H20 GPUs) under the same preconstructed preference dataset.
- Task-Arithmetic negatives outperform LLM-judge-relabeled and iterative negatives, which help only when mixed with the original preference data.
Strengths
Endogenous hard negatives are a clean conceptual contribution
Rejected responses come from the α=0.4 Task Arithmetic merge itself (§3.2), targeting the method's own failure mode without reward models or human labels; the base-model-negative control (Table 2: NP falls 0.9968→0.9497, Instruction 0.9952→0.8138) directly supports that the gain stems from merging-specific negatives rather than arbitrary negatives.
Strong headline result backed by a well-decomposed ablation
Table 1: 0.9968 avg NP vs Task Arithmetic 0.9538 and AdaMerging (Gen.) 0.9788; Table 2 isolates each component (positive-only SFT 0.9842, global coefficients 0.9822, both removed 0.9679), showing negative feedback and tensor-wise granularity are complementary.
Above-average evaluation fairness hygiene
Candid negative results on preference-data engineering
Efficiency claim measured under a controlled comparison
Weaknesses
Headline result is single-seed with no significance testing
Cost-to-peak metric embeds post-hoc checkpoint selection
Cross-backbone generalization is uneven
Iterative-data gains are max-of-sweep on the test metric
Core hyperparameters never varied
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (arXiv 2609.02273, cs.AI/cs.CL; Sun Yat-sen University, Zhejiang University, HiThink Research) proposes CoMerge, which reformulates multi-task LLM merging as preference optimization. Three central claims: (1) outputs of a naive Task Arithmetic (TA) merge act as free hard-negative responses, enabling preference-pair construction without human or reward-model annotation (§3.2); (2) DPO over tensor-wise merging coefficients — P×N = 1,445 scalars for Llama-3.1-8B-Instruct with five experts (§3.3) — mitigates parameter interference; (3) on MergeBench, CoMerge attains 0.9968 average normalized performance (NP), above all evaluated merging baselines and near Full-DPO (0.9983), at 60.1% lower GPU-resource cost to peak (Table 5). The evaluation spans three backbones (Llama-3.1-8B, Llama-3.2-3B, Gemma-2-2b-it), five task domains, and an unusually deep analysis of negative-sample construction (RQ3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CoMerge in one sentence?
CoMerge is a model-merging framework that treats merging as preference optimization: outputs of expert models serve as preferred responses, outputs of a naive Task Arithmetic merge serve as hard negatives, and DPO tunes only about 1,445 tensor-level merging coefficients instead of any model weights.
How well does CoMerge perform?
On MergeBench with Llama-3.1-8B-Instruct it reports 0.9968 average normalized performance — the best among the evaluated merging methods (Task Arithmetic 0.9538, AdaMerging 0.9788) and close to full-parameter DPO (0.9983) — using a single H20 GPU and 38.3 GPU-minutes to reach peak, versus four GPUs and 95.9 minutes for Full-DPO.
Does CoMerge need labeled data or human preference annotations?
No preference labels are needed: it uses about 1,000 raw prompts per expert task, sampled from the experts' training sets, plus the expert models themselves. Positive and negative responses are generated automatically by the expert and the naive merged model, so the preference signal is self-supervised.
Is CoMerge's code available?
The paper states that code, evaluation scripts, configuration files, and complete environment specifications will be released at github.com/Zheng-Jay/CoMerge upon publication; as of the September 2026 arXiv v1 they are promised but not yet available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are extracted as interleaved inline text, so visual evidence (the Figure 3 coefficient heatmap patterns, the Figure 2 pipeline diagram) could only be assessed via the surrounding prose, not the visuals themselves.
- Table formatting cues (bold/underline markings in Tables 1 and 8) are flattened by extraction; all numbers cited in this review come from the table cell values and prose, which appear intact through the Table 8 caption at the end of the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.