Instella-MoE Technical Report
Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Jialian Wu, Ximeng Sun, Wen Xie, et al.
cs.CL · cs.AI
Quick verdict
Instella-MoE is a genuinely fully open 16B MoE (2.8B active) trained on AMD GPUs with credible, verifiable fully-open SOTA claims, but headline margins are tiny, single-run, and error-bar-free.
Summary
Instella-MoE is a fully open Mixture-of-Experts language model with 16B total and 2.8B active parameters per token, trained from scratch on AMD Instinct GPUs. Its architecture adds two innovations: Gated MLA and FarSkip-Collective connectivity, which overlaps expert-parallel communication with computation. A multi-stage pipeline (7.1T-token pre-training, souped mid-training, 64K context extension, SFT, DPO, and two-stage RL with multi-teacher on-policy distillation) yields a base model averaging 76.7 on pre-training benchmarks, ahead of other fully open models (SmolLM3-3B 70.5; OLMo-3-7B 70.1) but behind Qwen3.5-4B-Base (79.5); the post-trained version averages 73.2 across twelve instruction-following, reasoning, math, coding, and chat benchmarks, which the authors rank first in the comparison, just above OLMo-3-7B-Think (71.97), although Qwen3.5-4B leads many individual benchmarks. An ablation shows an instruction-following RL expert lifts IFEval to 84.1 while regressing math, and the final multi-teacher on-policy distillation restores those gains (IFEval 83.7; average 75.7). The release includes stage-wise checkpoints, hyperparameters, data recipes, and code; pre-training is 12.7% faster and inference TTFT 39.2% better. Caveats: the headline architectural gains come from single 200B-token runs without variance estimates, margins over open-weight rivals are slim, and the excerpt shows no contamination audit of the pre-training corpus.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Instella-MoE-16B-A3B-Base, at 2.8B active parameters, is the strongest fully open base model in the comparison with a 76.7 average (Table 10), ahead of OLMo-3-7B (70.1), SmolLM3-3B (70.5), and OLMoE-1B-7B (61.9).
- The post-trained Think checkpoint's 73.2 twelve-benchmark average (Table 12) exceeds every compared fully open and open-weight model, including OLMo-3-7B-Think (71.97) and Gemma-4-E4B (70.47).
- Gated MLA improves the 200B-token ablation average from 49.86 to 50.33 and FarSkip-Collective preserves accuracy (50.38) while enabling a 12.7% pre-training throughput gain and 39.2% inference TTFT throughput gain.
- Disabling MoE load balancing (bias updates and auxiliary loss) during DPO prevents expert-routing drift (top-1 divergence 54% to 5%; Jaccard 0.42 to 0.94) and is necessary for DPO to help the MoE model.
- A two-stage RL recipe — IF-specialized RL followed by Multi-Teacher On-Policy Distillation — adds instruction-following capability (IFEval 77.1 to 83.7) without the math/reasoning regression of the IF expert alone.
Strengths
Strongest fully open base-model results, with internally consistent arithmetic
Table 10's 76.7 average recomputes exactly from the 13 printed column scores (sum 997.1), beating SmolLM3-3B (70.5), OLMo-3-7B (70.1), and OLMoE-1B-7B (61.9); it leads WinoGrande (86.5) and HumanEval+ (65.7) across all listed models, though it trails Qwen3.5-4B-Base (79.5) and Moonlight-16B-A3B is close (76.2).
Unusual release completeness
Checkpoints from every stage (Table 2), per-stage hyperparameters (Table 3), full data mixtures down to token counts per subset (Tables 4, 7, 22, 23, 24), curation prompts verbatim (Tables 18-21), and training code on AMD's Primus/Miles stacks make the release auditable to a degree few frontier-adjacent reports match.
MoE-specific post-training insight is genuinely novel and measured
Honest staging trade-off analysis
Non-CUDA infrastructure and efficiency results
Weaknesses
Headline architectural innovations rest on very thin evidence
Zero-mean framing overstates standing against open-weight peers
No variance quantification anywhere
Long-context baseline scores appear protocol-broken
The long-context cost of the final base model is a real capability regression
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Instella-MoE Technical Report — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Instella-MoE?
A fully open-source Mixture-of-Experts language model from AMD with 16 billion total parameters and 2.8 billion active per token, trained entirely from scratch on AMD Instinct MI300X/MI325X GPUs. Unlike open-weight releases, it ships weights from every training stage, the data recipes, training code (Primus/Miles), and evaluation protocols.
How does Instella-MoE compare to Qwen3.5-4B, OLMo-3, and Moonlight?
On the paper's own benchmarks, the base model averages 76.7 — above OLMo-3-7B (70.1), SmolLM3-3B (70.5), OLMoE-1B-7B (61.9), and Moonlight-16B-A3B (76.2), but below Qwen3.5-4B-Base (79.5). The post-trained Think model averages 73.2, the highest in its table, just above OLMo-3-7B-Think (71.97), while Qwen3.5-4B still wins several individual benchmarks (GPQA, AGIEval, AlpacaEval).
What are Gated MLA and FarSkip-Collective?
Gated MLA adds a small input-conditioned sigmoid gate to Multi-head Latent Attention that modulates per-channel attention outputs before the output projection. FarSkip-Collective is a connectivity scheme that lets expert-parallel all-to-all communication (Dispatch/Combine) overlap with attention and shared-expert computation by feeding layers slightly outdated or partial activations, raising pre-training throughput 12.7% and inference TTFT throughput 39.2% with no measurable quality loss in a 200B-token ablation.
Can I reproduce or use Instella-MoE?
Use: yes — checkpoints for every stage (pretrain, midtrain, base, SFT, DPO, Think) plus code and recipes are released on Hugging Face/GitHub under the AMD Instella project. Full retraining is not feasible outside large GPU clusters (7.1T tokens on MI300X/MI325X), but the release is unusually detailed, including exact hyperparameters, data mixtures, and even the LLM prompts used for SFT data curation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked '[...sampling gap...]' sections; content inside gaps (e.g., parts of the results narrative, Appendix A implementation prose, later references) could not be checked, so all absence claims in this review are scoped to the provided text.
- The final Instella-MoE-16B-A3B-Think row of Table 12 was cut off mid-cell by a sampling gap; the 73.2 average is taken from the abstract/introduction, and SFT (71.58), DPO (72.67), and stage rows are quoted from printed cells.
- No contamination/decontamination audit was visible in the provided text; it may exist in the full paper.
- AI-detect citation checks were not run; review is based solely on the embedded excerpt.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.