Higher-order pruning of experts in mixture-of-experts language models
Alex M. Tseng, Prannay Kaul, Luca Zancato, Wei Xia, Stefano Soatto
cs.LG · cs.AI
Quick verdict
HOPE extends MoE expert pruning to pairwise interactions; theory and experiments are strong, but gains are modest and Qwen eval anomalies temper the claims.
Summary
Mixture-of-Experts (MoE) language models must keep all experts in GPU memory even though each token activates few, motivating expert pruning. Existing methods (Frequency, EAN, MAN, REAP) score experts independently, ignoring that routing co-activates cooperative expert pairs. The paper derives HOPE, a second-order objective: from a calibration pass it builds a per-layer ExE interaction matrix F whose entries are gate-weighted output-norm products of co-selected expert pairs, and selects the prune-set by minimizing the binary quadratic form p^T F p under a per-layer budget, proven (Theorems 1-2) to minimize an upper bound on substitution error; REAP is recovered when off-diagonal terms are zeroed. Experiments span Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, and GLM-4.5-Air, two coding-focused calibration sets, six pruning rates, and the Tulu3 suite plus SWE-Bench Pro (3 trials each). HOPE attains the best overall average rank (2.07; top-1 in 39% of conditions), rank 1.58 at 50% pruning versus 2.42 for REAP, and up to +6.1% on agentic coding, with the margin growing at 40-50% rates. Off-diagonal terms average 33% of the diagonal and prune-sets are stable across trials (Jaccard >0.95). Main limitations: no significance testing; several decisive comparisons sit within trial noise; Qwen base evaluations show near-chance MMLU and ~0% PopQA, suggesting harness artifacts; and REAP wins GLM-4.5-Air language benchmarks at 40-50% pruning.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Expert-pruning error in MoEs is combinatorial, and a second-order objective over a per-layer pairwise interaction matrix F provably minimizes an upper bound on the substitution error (Theorems 1-2), with REAP as the special case that zeroes interaction terms.
- Across 54 conditions (3 MoEs up to 122B, 2 calibration sets, 6 pruning rates, Tulu3 + SWE-Bench Pro), HOPE achieves the best average rank (2.07 overall; 1.58 at 50% pruning vs 2.42 for REAP) and a 73% head-to-head win rate.
- HOPE's advantage grows with pruning rate and is largest on agentic coding, code generation, and compositional reasoning (up to +6.1% on SWE-Bench Pro at 50%), consistent with preserving cooperative expert structure.
- The interaction structure HOPE exploits is stable and real: prune-sets are reproducible across trials (Jaccard >0.95), off-diagonal terms average 33% of the diagonal, and HOPE's edge over REAP persists after 1B tokens of SFT (Sec 4.5, Fig 6).
Strengths
Clean theoretical framing that provably subsumes the SOTA baseline
Theorems 1-2 derive an upper bound on substitution error minimized by a binary QP over an ExE interaction matrix; Figure S4 shows the F-diagonal recovers REAP's rankings (Spearman 0.988) and zeroing off-diagonals empirically reproduces REAP's prune-set (Sec 3.5), so any gain is directly attributable to interaction terms rather than confounded implementation differences.
Unusually broad and transparent empirical campaign
54 conditions (3 models up to 122B, 2 calibration sets, 6 pruning rates, Tulu3 + SWE-Bench Pro) with 3 trials each reporting mean+-std; the paper discloses HOPE's own failures (last place in 1/54 conditions, worst deficit -4.5%) alongside REAP's 5 last-place finishes and Freq's 43 (Sec 4.2).
Empirical evidence aligned with the claimed mechanism
Deployment-relevant cost accounting and honest negative results
Weaknesses
Evaluation validity problems on Qwen models contaminate part of the evidence base
Rank-averaged headline claims mask systematic per-condition losses
No statistical inference despite variance being reported
Theory-practice gaps weaken the 'provable' framing
Missing standard sanity checks for a compression paper
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HOPE (Higher-Order Pruning of Experts)?
HOPE is a method for shrinking Mixture-of-Experts (MoE) language models by permanently deleting whole expert MLPs. Unlike earlier methods that score each expert independently, HOPE first runs a calibration pass to record how often pairs of experts are co-activated and how strongly they jointly contribute, then solves a small quadratic program per layer to pick which experts to remove. The paper proves this minimizes an upper bound on the error introduced by pruning, and it generalizes the earlier REAP method (REAP is the case where pairwise interactions are ignored).
How much better is HOPE than REAP?
According to the paper, HOPE has the best average rank across 54 experimental conditions (3 models, 2 calibration sets, multiple benchmarks). At aggressive 50% pruning, HOPE's average rank is 1.58 out of 5 methods versus 2.42 for REAP, with the largest single gain (+6.1%) on agentic coding (SWE-Bench Pro). However, it is not uniformly better: REAP wins on GLM-4.5-Air language benchmarks at 40-50% pruning, and another baseline (EAN) beats HOPE in one high-pruning agentic setting, so the advantage is statistical-average rather than universal. Calibration costs only ~6-7% more than REAP.
Does pruning 50% of experts hurt model performance?
At high pruning rates most methods degrade, especially on knowledge and recall tasks (e.g., PopQA, TruthfulQA drop sharply in the reported tables), while coding and reasoning degrade less. The paper also reports cases where pruned models match or beat the unpruned model, attributing this to calibration acting as implicit task specialization; an independent review notes that some of these base-model scores look anomalous (e.g., near-chance MMLU that doubles after pruning), so those particular gains should be treated with caution.
Is the HOPE code and data available?
The paper's reproducibility statement says code for calibration, QP solving, and analyses will be released upon publication, citing a GitHub path (awslabs/hybrid-model-factory, examples/research/HOPE). All models (Qwen3.5-35B/122B, GLM-4.5-Air), calibration datasets (Evol-CodeAlpaca, SWE-bench Verified trajectories), and benchmarks (Tulu3 suite, SWE-Bench Pro) are public, and Appendix D details solver settings, hyperparameters, and evaluation configurations.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 and S1-S7 could not be numerically inspected from text extraction; plotted claims (e.g., Figure S1 head-to-head statistics, Figure 1 rank curves) were cross-checked only against Tables S1-S6 where possible.
- Mathematical notation (norms, sums, superscripts) may be garbled by extraction; derivations were assessed on structure and logic rather than symbol-by-symbol fidelity.
- This is a v1 preprint (arXiv:2609.18916); the review reflects the embedded text only, and the promised code release could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.