ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs
Zukang Xu, Zhixiong Zhao, Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, Dawei Yang
cs.AI
Quick verdict
ACE skips MoE experts via two weight-derived views, calibration-free; strong results on 3 models, but single-run stats and partial baselines temper claims.
Summary
The paper introduces ACE, a training-, calibration-, and checkpoint-preserving framework that adaptively skips expert slots in Mixture-of-Experts LLMs at inference. ACE combines two offline, parameter-derived estimates: Global Spectral Proxy (GSP), which scores each SwiGLU expert's transformation capacity from its coupled gate/up/down projections and RMSNorm scaling, and Router-Conditioned Refinement (RCR), which measures each expert's response along a direction prototype built from centered router weights. Online, a slot is skipped only when both normalized views rank it below a global threshold, with the top-1 expert always retained and surviving gates renormalized; inference adds only table lookups and scalar operations. Across Qwen3-30B-A3B-Instruct-2507, Qwen3.6-35B-A3B, and Gemma-4-26B-A4B-it, at 10–60% realized skipping on eight benchmarks, ACE generally achieves the best accuracy–perplexity trade-off, most visibly at 40–60% skipping (e.g., at 50% on Qwen3.6, PPL 8.67 vs MoDES 9.42 and accuracy 75.57 vs 71.42). On an A100 the authors report up to 2.25× prefill and 1.41× decode speedups. Limitations: results are single deterministic runs (seed 42) with no variance or significance testing; several strong baselines (Top-P, ExpertSparsity, AIMER) appear on only one of the three models; Gemma perplexity is omitted; and RCR adds modest gains over GSP alone in several settings.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GSP and RCR estimate expert contribution from frozen weights alone, with no calibration data, training, or checkpoint modification (Secs. 3.1–3.2).
- Skipping a slot only when both normalized views agree, while always retaining top-1, yields the best accuracy–PPL trade-off across three MoE models at 10–60% realized skipping, with the largest margins at 40–60% (Tables 1–3).
- At 50% skipping on Qwen3.6-35B-A3B, ACE improves WikiText-2 PPL by 7.96% and average accuracy by 4.15 points over MoDES, the strongest baseline reported for that model (Table 19).
- ACE realizes up to 2.25× prefill and 1.41× decoding speedups on an A100, with online overhead limited to table lookups and scalar operations (Table 7).
- The dual-view rule provably removes no more router mass than either single view under the same threshold, with perturbation bounded by 2αF_max (Appendix A.2, Eqs. 60–71).
Strengths
Theory-to-proxy chain is unusually complete
Appendix A.2 derives the SwiGLU norm bound motivating GSP (Eqs. 21–29), proves shared-translation invariance of router weights justifying centering (Eqs. 30–38), obtains the Gaussian-tilting mean shift ≈ Σ·w̄ motivating RCR (Eqs. 39–49), and proves the max rule equals the intersection of single-view candidate sets with skipped-mass dominance and perturbation bound ∥y−ŷ∥ ≤ 2αF_max (Eqs. 50–71).
Controlled, disclosed evaluation
Matched realized (not requested) skipping budgets across methods (Sec. 4.1), complete 10–60% curves on three models (Tables 1–3), and full task-level granularity for every cell (Tables 9–26) make the comparison auditable.
Deployment-relevant measurement
Conservative safeguards are explicit and ablated
Weaknesses
No statistical uncertainty anywhere
Baseline coverage differs per model
RCR's incremental value over GSP alone is small in several cells
Gemma-4 perplexity omitted and efficiency scope narrow
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ACE (Secs. 1, 3) targets token-adaptive expert-slot skipping in pretrained MoE LLMs without training, calibration data, or checkpoint edits. Its claims: (i) router gates alone conflate routing preference with expert contribution; (ii) two weight-only estimators — GSP, a spectral surrogate of SwiGLU capacity (Sec. 3.1), and RCR, a router-direction response probe (Sec. 3.2) — jointly fix this; (iii) conservative max-fusion with top-1 retention delivers the best quality at matched realized skip budgets, with up to 2.25×/1.41× prefill/decode speedups. Scope: Qwen3-30B-A3B-Instruct-2507, Qwen3.6-35B-A3B, Gemma-4-26B-A4B-it; eight benchmarks; 10–60% realized skipping.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ACE for Mixture-of-Experts models?
ACE (Adaptive Calibration-Free Expert Skipping) is an inference-time method that dynamically skips low-contribution expert slots in MoE LLMs. Each routed expert is scored using two statistics computed offline from model weights only — a spectral proxy of its SwiGLU transformation capacity (GSP) and its response along a direction derived from centered router weights (RCR). A slot is skipped only when both views rate it low, and the top-1 expert is always kept. No calibration data, retraining, or checkpoint changes are needed.
How much faster is ACE and how much accuracy does it lose?
The authors report up to 2.25× faster prefill (TTFT) and 1.41× faster decoding (TPOT) on an NVIDIA A100 at 60% skipping. Quality stays closest to the unpruned model among tested methods across 10–60% skip budgets on three MoE models; for example, at 50% skipping on Qwen3.6-35B-A3B it reaches 75.57 average accuracy vs 71.42 for the best baseline reported there, with WikiText-2 perplexity 8.67 vs 9.42.
Does ACE need training or calibration data?
No training or labeled data. GSP and RCR tables are computed from frozen weights and router geometry. The only data-dependent step is one unlabeled pass used to map a target skipping percentage to a score threshold; the paper shows these thresholds are stable across different workloads (Appendix B.1), suggesting they can be reused across datasets.
Is code available for ACE?
The paper links to github.com/xzktx003/moe-prune. The evaluated models (Qwen3-30B-A3B-Instruct-2507, Qwen3.6-35B-A3B, Gemma-4-26B-A4B-it) and the evaluation stack (EvalScope 1.4.1, greedy decoding, seed 42) are public, but reproducing the 26–35B BF16 experiments requires substantial multi-GPU resources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 appears only as garbled axis/legend text in the extraction; its plotted curves could not be assessed — accuracy-efficiency trend judgments rely on the result tables instead.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.