SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, Xuehai Zhou
cs.OS · cs.AI
Quick verdict
Claims near-full-load MoE offloading via seq2seq prediction, matroid prefetching, CUDA-graph runtime; strong results but lacks variance, code, key baselines.
Summary
SeqMoE is a MoE offloading system for memory-constrained LLM inference that aims to approach full-load (all-weights-on-GPU) performance. It combines four components: (i) a seq2seq expert-activation predictor built on Mamba2 that forecasts multi-step, multi-layer activations from the full layer stack — the authors claim the first application of sequence modeling to this problem; (ii) prefetch scheduling formulated as job sequencing with deadlines and solved by matroid greedy over confidence- and deadline-tagged tasks; (iii) a probabilistic Belady eviction policy driven by recursive 8-step forecasts; and (iv) a graph-compatible runtime using shared fixed-base cache slots, fused MoE kernels, and four synchronization-free orchestration disciplines enabling end-to-end CUDA Graph capture. Evaluated on Qwen3-30B-A3B, Qwen3.6-35B-A3B, GPT-OSS-120B, and DeepSeek-V4-Flash across RTX 4090/5090/RTX PRO 6000 GPUs against llama.cpp, MoE-Infinity, KTransformers, and FreeToken, it reports 96.97% average hit rate and 80.22% of full-load decode performance at 45% expert residency versus 88.50% and at most 51.50% for the strongest baselines. Limitations: no variance or statistical reporting, no code release, no end-to-end comparison against the prefetching systems it critiques (ProMoE, MoE-APEX), and a headline average that cannot be exactly re-derived from Figure 6.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Expert activations carry sequence-level dependence strong enough that a Mamba2 seq2seq predictor reaches 85.81–90.90% top-(k+3) one-step recall across four models, with 8-step recursive forecasts losing only 3.58 percentage points on average (Section 9.4).
- Formulating prefetch scheduling as Job Sequencing with Deadlines and solving it with matroid greedy beats independent per-task scoring (MoE-Infinity-style) and EDF-lexicographic ordering (ProMoE-style), especially at small cache capacity (Sections 3.2, 6, 9.3).
- Forecast-driven probabilistic Belady eviction outperforms history-based LRU/LFU/PrefillHot, reaching a 96.97% average hit rate at 45% residency versus 88.50% for LRU, the strongest baseline (Sections 7, 9.3).
- Compute-transparent expert placement (shared fixed-base cache slots) plus four synchronization-free orchestration disciplines enable end-to-end CUDA Graph capture of the entire offloading pipeline (Section 8).
- Combined, SeqMoE achieves 80.22% of full-load decode performance at 45% expert residency, versus at most 51.50% for FreeToken and 28.03% for other baselines (Section 9.2).
Strengths
Novel predictive core with a clean falsification test
Fig 2 trains decoder-only Transformers on original vs step-shuffled activation sequences: convergence holds on ordered data (recall ~92% in Fig 1c) and collapses under shuffling (~23%), supporting the claim of learnable sequence-level dependence. The production Mamba2 predictor costs only 3.46% of execution time and 112–228 MB memory (Section 9.5).
Principled, provably-grounded prefetch scheduling
Section 3.2 casts prefetch selection as a weighted matroid over deadline-feasible task sets; Table 2 gives a concrete instance where both MoE-Infinity's confidence-decay score and ProMoE's EDF-lexicographic order miss the optimum (A,B,D,E). Algorithm 1 turns this into an implementable min-heap greedy with EMA-adapted per-layer transfer budgets.
Ablation-supported cache design
Comprehensive systems evaluation
Real runtime engineering, honestly costed
Weaknesses
No statistical treatment anywhere
Headline average not re-derivable
Missing end-to-end baselines
Underspecified training and no sensitivity/OOD analysis
No artifact release; stack heterogeneity muddies the ratio
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SeqMoE (arXiv 2609.12978, cs.OS) targets single-GPU MoE inference under device-memory constraints: experts live in host DRAM and only 15–45% of cache slots reside on the GPU. The claim chain is: expert activations exhibit sequence-level dependence (Fig 2) → a Mamba2 seq2seq predictor provides long, accurate forecast windows (Seq-1 top-(k+3) recall of 90.90%/88.39%/90.76%/85.81% on QW3/QW36/GPT/DSV4; 8-step recursion costs only 3.58pp, Section 9.4) → forecasts drive joint prefetch scheduling (Job Sequencing with Deadlines solved by matroid greedy, Section 6) and probabilistic Belady eviction (Section 7) → a synchronization-free runtime captures the entire pipeline in CUDA Graphs (Section 8). Headline: 96.97% average hit rate and 80.22% of full-load decode performance at 45% residency, versus 88.50% (LRU) and ≤51.50% (FreeToken) for baselines (Sections 9.2–9.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does SeqMoE solve?
It lets large Mixture-of-Experts language models run on GPUs whose memory cannot hold all weights. Experts are kept in CPU RAM, and the system predicts which experts each upcoming step needs, prefetches them over PCIe, and manages a GPU expert cache — aiming to run almost as fast as if every weight were on the GPU (full-load performance).
What performance does SeqMoE achieve?
With 45% of experts resident on the GPU, the paper reports an average 96.97% expert hit rate and 80.22% of full-load decode speed across four MoE models (Qwen3-30B-A3B, Qwen3.6-35B-A3B, GPT-OSS-120B, DeepSeek-V4-Flash), versus 88.50% hit rate and at most 51.50% of full-load for the strongest compared systems (LRU caching; FreeToken).
How is SeqMoE different from prior MoE offloading systems?
Three ways: (1) it predicts expert activations with a seq2seq sequence model (Mamba2) that forecasts many future steps at once, instead of shallow cross-layer guesses or first-order statistics; (2) it schedules prefetches jointly under bandwidth and deadline constraints using a matroid-greedy algorithm with provable optimality; (3) its runtime is CUDA Graph-compatible — the whole pipeline (prediction, caching, miss loading) runs without CPU-GPU synchronization, unlike prior systems that break graph execution.
Is the SeqMoE code available?
The paper does not mention a public code, model, or trace release. It describes an implementation built on Hugging Face Transformers with hooks, custom CUDA kernels for cache management and miss loading, and specific NVIDIA GPUs (RTX 4090, RTX 5090, RTX PRO 6000).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 6–9 were extracted as unlabeled numeric runs; per-bar baseline attribution was cross-checked against in-text percentages where possible (decode-throughput ratios verified for all four models), but ambiguous panels (e.g., TTFT attribution) were not cited numerically in this review.
- Mathematical notation (e.g., the CacheScore formula in Section 7.1 and indexing in Section 5.2) is garbled by PDF extraction; formulas should be read from the original PDF.
- The provided text ends cleanly at reference [45] with no signs of truncation; body sections 1–10 and the conclusion are fully present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.