ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Ahin Lee, Sehyun Yun, Joonha Park, Taesik Gong
cs.LG · cs.AI · cs.CL
Quick verdict
Groups redundant expert LoRAs into shared higher-rank adapters for MoE PEFT; solid 4-backbone evidence and ablations, but gains lack significance testing.
Summary
The paper identifies three fragmentations in expert-wise LoRA for MoE fine-tuning: capacity is split into narrow low-rank updates, gradient supervision is sparse under routing, and execution degrades into many small GEMMs. Observing that expert adapters become functionally similar during fine-tuning (measured by CKA on probe activations), ACE groups similar experts per layer via Louvain community detection and replaces their adapters with one shared higher-rank LoRA under the same trainable budget, initialized from a CKA-central anchor after a 150-step warm-up. Grouped adapter execution computes each shared adapter once per active group for gate/up projections, where routed experts share inputs. Across four MoE backbones (OLMoE, Qwen1.5-MoE, Moonlight, Kimi-Linear) and 12 benchmarks, ACE reports the best mean accuracy among parameter-matched PEFT baselines on the three backbones with full baseline coverage (e.g., OLMoE 75.61 vs LoRA 74.11), plus 1.31-1.48x wall-clock training speedups over expert-wise LoRA at unchanged peak memory. Ablations attribute gains to rank consolidation, CKA grouping, and grouped execution; full merging hurts (73.57 vs 74.11). Key limitations: no significance tests for modest margins, the largest backbone is compared only against LoRA, grouped execution excludes down projections, and the default Louvain resolution is not the sweep optimum.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Expert-wise LoRA in MoE models fragments adaptation along three coupled dimensions: capacity, gradient supervision, and execution (Section 3.1).
- Expert LoRA adapters become functionally similar during fine-tuning, and CKA-based grouping reliably identifies interchangeable adapters (Figure 2 swap analysis, Figure 5).
- Consolidating similar experts into shared higher-rank adapters under the same trainable budget yields the highest observed mean accuracy among parameter-matched PEFT methods on the three backbones with complete baseline coverage (Table 1).
- Grouped adapter execution gives 1.31x-1.48x wall-clock training speedup over expert-wise LoRA without increasing peak memory, applicable to gate/up but not down projections (Table 2, Limitations).
- Consolidation granularity matters: full merging underperforms expert-wise LoRA (73.57 vs 74.11), so similarity-guided grouping is necessary (Table 5).
Strengths
Clear problem decomposition with matched diagnostics
Section 3.1 formalizes capacity, gradient, and execution fragmentation, and Section 5.1 tests each with a dedicated metric: effective rank (LoRA-r2 mean 1.77 vs ACE close to LoRA-r64's 46.8 at 1/33 the params), gradient CV 1.449 to 0.531 and zero-grad ratio 3.18% to 1.17% (OLMoE up_proj), and CUDA profiling (mm calls -41.3%, Table 4).
Unusually complete ablation suite
Table 5 isolates all three components: Random grouping 74.72, Full Merge 73.57 (worse than LoRA's 74.11), w/o Rank Cons 72.63, w/o Grouped Exec identical accuracy but 1.89 to 2.83 h. The honest inclusion of a configuration that underperforms the baseline (Full Merge) is rare.
Broad, seed-consistent evaluation
Well-instrumented efficiency claims
Thorough robustness analysis
Weaknesses
No statistical significance testing
Largest backbone lacks MoE-aware baselines
CKA grouping's marginal contribution is modest
Default hyperparameter not the sweep optimum, selection rule informal
Efficiency compared only against expert-wise LoRA
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ACE is a PEFT method for MoE LLMs. The paper argues that expert-wise LoRA (one rank-r adapter per expert) causes three fragmentations - capacity (rank shrinks as 1/E under a fixed budget, Sec. 3.1.1), gradient supervision (each adapter sees only its routed token fraction, 3.1.2), and execution (many small GEMMs, 3.1.3) - and that this separation is often unnecessary because expert adapters converge to functionally similar groups (CKA heatmaps, Fig. 2; swap test, Fig. 5). Claims are scoped carefully: the abstract asserts the best mean accuracy only *on the three backbones with complete baseline coverage*, and speedups only *against expert-wise LoRA*. Central claims: (1) CKA-guided consolidation improves accuracy at equal parameter budget, (2) grouped execution speeds training 1.31-1.48x without memory increase, (3) consolidation granularity matters, since Full Merge underperforms LoRA (73.57 vs 74.11).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ACE and how does it improve LoRA fine-tuning of MoE models?
ACE groups experts whose LoRA adapters behave similarly (measured by CKA similarity on probe activations) and replaces each group's expert-specific adapters with one shared, higher-rank adapter under the same trainable-parameter budget. It also computes each shared adapter once per group instead of per expert, turning many small matrix multiplications into fewer large ones. The paper reports higher accuracy than parameter-matched PEFT baselines on three MoE backbones and 1.31x-1.48x faster training than expert-wise LoRA.
How much faster is ACE than LoRA, and does it use more memory?
The paper reports wall-clock training speedups of 1.31x (OLMoE), 1.48x (Qwen1.5-MoE), 1.42x (Moonlight), and 1.45x (Kimi-Linear) over expert-wise LoRA, with peak GPU memory essentially unchanged (e.g., 98.18 vs 98.21 GB on OLMoE). The one-time consolidation overhead is at most 1.83% of total training time.
What are the limitations of ACE?
Grouped execution applies only to gate/up projections (down-projection inputs are expert-specific); it assumes adapter redundancy, so over-consolidation can hurt - the full-merge ablation scores 73.57 vs LoRA's 74.11; the Louvain resolution may need tuning when expert specialization matters; and in the paper the largest backbone (Kimi-Linear, 48B) is compared only against LoRA.
Is the ACE code available?
The paper states code is available at github.com/UbiquitousAILab/ACE, and the appendices provide detailed model configurations, hyperparameters, seeds, and dataset statistics intended to support reproduction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's heatmap renders as repeated placeholder glyphs in the extracted text; individual CKA cell values were not readable, though the caption and surrounding analysis are intact.
- Figure 7 (rank sweep) values are presented only graphically; exact per-rank accuracies are not recoverable from the extraction, though the stated trend is described in text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.