A Distributional Optimisation Perspective on Combining Models in Deep Learning
Congye Wang, Yan Lin, Zheyang Shen, Matthew A. Fisher, Chris. J. Oates
cs.LG · stat.ML
Quick verdict
Casts ensembles and LoRA averaging as entropy-regularised distributional optimisation; convexity only for ensembles. Honest, sound study with modest gains.
Summary
The paper casts two standard model-combination strategies — output ensembles and LoRA adapter averaging — as entropy-regularised optimisation over probability distributions on parameter space. It proves the ensemble objective is convex (strictly so with entropy regularisation), so mean-field Langevin dynamics (MFLD) convergence guarantees apply, while the LoRA-averaging objective is generally non-convex because the network is nonlinear in the averaged adapter. Methods compared: MFLD, variational gradient descent (VGD, applied to LoRA averaging here for the first time), and a novel functional variant (FVGD). On capacity-limited Spiral and MNIST classification, jointly trained ensembles beat independently trained ones and match a larger single network. On Llama-3-8B fine-tuned over eight commonsense benchmarks, VGD reaches 85.48% mean accuracy at rank 32 versus 84.97% (Standard AdamW) and 85.16% (MFLD), and 83.54% at rank 256 versus 82.75%/81.83%, with paired bootstrap CIs excluding zero; VGD-based PoC merging reaches 87.31%, and joint 8-branch training 89.33% at 8x training and inference cost. FVGD underperforms, and the authors diagnose this honestly (near-orthogonal Jacobian pull-back). Main limitations: proof-of-concept scope, and single-seed lambda/rank/model ablations.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Lifted to distributions over parameters, ensembles and LoRA averaging yield distinct objectives: the ensemble objective is convex (strictly with entropy regularisation) while LoRA averaging is generally non-convex, so MFLD convergence guarantees transfer only to ensembles.
- Jointly trained ensembles outperform independently trained ensembles on capacity-limited classification tasks and match a single larger network.
- VGD applied to LoRA averaging outperforms standard AdamW fine-tuning and MFLD on Llama-3-8B commonsense tasks, most clearly at rank 256 (83.54% vs 82.75%/81.83%), at ~6% wall-clock overhead over MFLD.
- Entropic regularisation strength has negligible effect at moderate values but degrades performance sharply at large λ, with sensitivity that grows as model size shrinks.
- The functional variant FVGD gives no consistent advantage because the Jacobian pull-back is nearly orthogonal to the intended function-space update.
Strengths
Crisp convexity separation between the two combination strategies
The Proposition in §2.2 proves the ensemble loss (Eq. 4) is convex and strictly convex under entropy regularisation, while LoRA averaging (Eq. 5) is non-convex because f is nonlinear in the averaged BA; the paper is explicit that MFLD convergence guarantees (Chizat 2022) transfer only to the former. Appendix B cleanly equates KL and entropy regularisation via a confining potential.
Exemplary handling of a negative result
FVGD, the paper's novel method, underperforms, and §E.7 diagnoses why: pull-back cosine similarities of 0.002–0.012 across six configurations, and a regularised inverse pull-back that fixes alignment (cosine ~0.875) but yields parameter norms ~7.1e-7 versus supervised gradient norms ~1.14e-1. Reported openly in Finding 3 (§3.1) rather than buried.
Statistically disciplined headline comparison
Thorough sensitivity analysis
High protocol transparency
Weaknesses
Single-seed ablations carry the secondary conclusions
No compute-matched control for the joint-training headline
Theory applies where experiments matter least
Modest absolute gains at matched budget
Missing LoRA-ensemble baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.24328, cs.LG/stat.ML; Newcastle University) proposes viewing model combination — output ensembles (Eq. 1) and LoRA adapter averaging (Eq. 2) — as entropy-regularised optimisation over distributions on parameter space, J(Q) = L(Q) + λE(Q) (Eq. 3). Its central claims: (i) the ensemble loss (Eq. 4) is convex, strictly so under entropy regularisation (Proposition, §2.2), so mean-field Langevin dynamics (MFLD) convergence theory (Chizat 2022) applies; (ii) the LoRA-averaging loss (Eq. 5) is generally non-convex because f is nonlinear in the averaged BA, so no such guarantees hold; (iii) VGD (Wang & Liu 2019) and a novel functional variant FVGD are assessed against MFLD on synthetic classification and Llama-3 LoRA fine-tuning. The authors explicitly disclaim state-of-the-art ambitions (§1.2), which is appropriate scoping for a framework-and-evidence paper.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper actually propose?
It treats the set of models being combined (an ensemble, or a set of LoRA adapters that get averaged) as a probability distribution over parameter space, and optimises that distribution directly with an entropy regulariser. This unifies training and combination: a convexity proof shows convergence theory applies cleanly to ensembles but not to LoRA averaging.
Does the proposed VGD method beat standard LoRA fine-tuning?
Modestly, in their experiments. On Llama-3-8B across eight commonsense benchmarks, VGD averages 85.48% accuracy at rank 32 versus 84.97% for standard AdamW (8 seeds, bootstrap CI excluding zero), and 83.54% versus 82.75% at rank 256. Larger gains come from training 8 adapters jointly (89.33%) but that costs 8× training and inference.
What is FVGD and did it work?
FVGD is the paper's novel function-space variant of variational gradient descent, running the kernel method on model outputs instead of parameters. It underperformed parameter-space methods in their tests; the authors show the Jacobian pull-back is nearly orthogonal to the intended update (cosine similarity ~0.002-0.012), a candid negative result.
Can I reproduce the experiments?
The classification experiments (Spiral, MNIST) need only a single GPU and are fully specified (architectures, seeds, step sizes in §D.1/C). The LLM experiments require 8×H100 GPUs; hyperparameters, seeds and evaluation details are given in §D.2, and the paper references a custom PEFT fork 'released with code', though no repository URL appears in the text.
How does this relate to model merging / model soups?
It is a complementary, principled angle: instead of combining already-trained checkpoints with fixed or learned weights, it jointly trains the collection of models so that the combined predictor's loss is the optimisation target. The paper's PoC-merge experiments (Table 1 block 3) show merging 8 rank-32 adapters beats training one rank-256 adapter directly, consistent with prior work by Nitanda et al. (2025).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 appear in the extracted text only as axis labels and captions; visual trend judgments rest on the reported numbers and caption text.
- No code repository URL is visible in the extracted text; codeAvailable is inferred from the phrase 'PEFT custom fork (commit released with code)' in Section C.5.
- Tables 5 and 7 contain OCR-style artefacts (e.g. '79`64', stray characters in the MFLD seed-2026 row); a few individual cell values were treated as approximate, though all block means quoted are consistent across tables.
- Reference list includes items dated 2025-2026; no external verification of citations was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.