ESTS at WMT26: Routing-Informed Expert Pruning for Model Compression
Liu O. Martin, Lucas Bandarkar, Nanyun Peng
cs.CL · cs.AI · cs.LG
Quick verdict
Clear, honest WMT26 system paper: GPT-OSS-20B pruned to 4.2–7.8B translation specialists via routing mass; only internal pseudo-reference eval, no ablations.
Summary
This WMT26 Model Compression shared-task paper describes six translation systems (team ESTS) for English→Simplified Chinese and English→Egyptian Arabic, all derived from GPT-OSS-20B (20.915B parameters). The authors rank mixture-of-experts experts by task-specific routing mass measured via vLLM hooks, allocate per-layer retained capacity proportional to Jensen–Shannon routing divergence between target-language and English prefill distributions, and physically remove low-scoring experts. Pruned specialists are recovery-tuned on 43,997 (ZHO) or 9,999 (ARZ) synthetic translation pairs that GPT-5.1 generated from a 44,000-example English corpus spanning news, social, and speech sources, then further compressed with MXFP4 quantization of retained expert projections. Submitted models span 4.186B–7.770B parameters (62.8%–80.0% reduction) and 4.55–6.33 GiB packed. Internal xCOMET-XL scores on 198 WMT26 blindset documents, using GPT-5.1 pseudo-references, show ZHO drops of 0.0179–0.0469 from the base (0.4309) but ARZ gains at milder operating points (k22: +0.0126; k24: +0.0068). A robust inference stack handles category inference, output validation, retries, segmented fallback, and deterministic JSON reconstruction, with no hard errors reported. The paper is unusually candid: it notes that active-expert computation is unchanged (no speedup claim), that packed size is not VRAM, and that pseudo-reference circularity may bias scores. Main limitations: no significance testing, no ablations isolating pruning, recovery tuning, or quantization, and official WMT26 results still pending.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Task-specific routing mass plus JS-divergence-based layerwise capacity allocation permits removing 62.8%–80.0% of GPT-OSS-20B's parameters (down to 4.186B–7.770B) while retaining usable translation quality after recovery SFT.
- Recovery fine-tuning on GPT-5.1 synthetic data allows the moderate-compression ARZ specialists (k22, k24) to exceed the uncompressed base on internal xCOMET-XL (+0.0126 and +0.0068).
- MXFP4 post-training quantization of retained expert projections shrinks packed artifacts to 4.55–6.33 GiB with no degradation reported on internal scores.
- A deterministic inference stack (category inference, validation, retries, segmented fallback, source-owned JSON reconstruction) produced a valid output for every evaluated blindset input with no hard errors.
Strengths
Candid self-assessment of evidence limits
Section 5 and the Table 3 caption explicitly state that the base comparison does not isolate pruning, that GPT-5.1 pseudo-references may bias scores toward its own output style, that active-expert MLP computation is unchanged (no decoding speedup claimed), and that packed sizes measure storage, not peak VRAM.
Unusually complete algorithmic specification
Appendix A gives the exact JS-divergence formula and discloses the deviation from Bandarkar et al. (2026) entropy normalization; Appendix B specifies budget-constrained proportional allocation with capping and Hamilton rounding; Section 3.1 states calibration sizes (660 examples/direction, 4,400 ZHO / 9,999 ARZ divergence pairs).
Moderate-compression ARZ systems beat the uncompressed base internally
Robust, concretely specified inference engineering
Precise failure accounting
Weaknesses
No statistical treatment of evaluation differences
No ablations isolating the components
Circular, internal-only quality evidence
No efficiency measurements despite the compression framing
No comparison against alternative compression systems
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a shared-task system report for the unconstrained WMT26 Model Compression task, describing six submissions (team ESTS; three per direction: EN→ZHO, EN→ARZ), all derived from GPT-OSS-20B (20.915B parameters; E=32 experts, K=4 active per MoE layer). Central claims: (1) task-specific routing mass plus JS-divergence-driven layerwise capacity allocation enables expert removal of 62.8%–80.0% (Table 2: 4.186B–7.770B parameters); (2) recovery SFT on GPT-5.1 synthetic data restores quality, with moderate-compression ARZ systems exceeding the base internally (Table 3: k22 +0.0126, k24 +0.0068); (3) MXFP4 quantization of retained expert projections yields packed sizes of 4.55–6.33 GiB; (4) the deterministic inference stack produced no hard errors on the 198-document blindset (Section 5). Claims are consistently and explicitly scoped — the authors repeatedly disclaim official results, speedup, and causal attribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is routing-informed expert pruning?
In mixture-of-experts (MoE) language models, a router activates only a few experts per token. This method records how much routing weight each expert receives when the model processes translation-like text (called routing mass), ranks experts by that score within each layer, and physically deletes the lowest-ranked ones. How many experts each layer keeps is decided by the Jensen–Shannon divergence between the target-language and English routing distributions, so layers that look more language-specialized retain more experts.
How much smaller are the ESTS WMT26 models than GPT-OSS-20B?
The six submitted systems range from 4.186B to 7.770B parameters, down from the 20.915B base — parameter reductions of 62.8% to 80.0%. After MXFP4 quantization of the retained expert weights, packed checkpoint sizes are 4.55 to 6.33 GiB, versus about 39 GiB for the base model in BF16.
Did the compressed models lose translation quality?
On the authors' internal xCOMET-XL evaluation (198 WMT26 blindset documents scored with GPT-5.1 pseudo-references), all three Chinese-direction systems scored below the base model (0.4309), by 0.0179 to 0.0469 points. The Arabic systems at milder compression slightly exceeded the base: k22 scored +0.0126 and k24 +0.0068 above it, while the most aggressive Arabic setting (k26) dropped 0.0199. These are internal scores; official WMT26 results were pending at publication.
Is the code or the pruned models available?
The paper does not state any code or checkpoint release. However, it documents the full recipe: the data sources (WMT News Crawl 2025, Webis-TLDR-17, Bluesky, SPoRC), GPT-5.1 labeling settings, the pruning and capacity-allocation formulas (Appendices A and B), LLaMA-Factory fine-tuning hyperparameters, and NVIDIA ModelOpt quantization.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.