The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu, Xiaodong Zeng
cs.AI
Quick verdict
Edge0 serves a 35B MoE from SSD at 20 tok/s inside 3 GiB via one-token-ahead routing prediction plus an unmerged recovery LoRA; solid, unusually honest systems work, but headline gains lack external offloading baselines.
Summary
Edge0 is a streaming Mixture-of-Experts inference engine that serves 35B-class models from SSD on consumer hardware. Because a 19.5 GB int4 checkpoint crowds out a 24 GB machine, expert weights stay on disk, are mmap-streamed per layer, and peak memory tracks the active set (2.9 GiB). To hide disk latency, a per-layer 'prerouter' head predicts the next layer's expert routing one token ahead; the prediction is consumed as the routing itself, so the staged expert set exactly equals the routed set and nothing is dropped. An unmerged recovery LoRA, distilled on the student routing path, recovers quality lost to int4 quantization and routing replacement (merging then re-quantizing erases most of the delta: 18% logits-level retention). On a Mac mini M4 Pro, a 35B tier (Qwen3.6-35B-A3B at K=4) decodes at 20.4 tok/s; an 8B MLA+MoE tier reaches 28 tok/s in 1.5 GiB. OpenCompass gaps vs fp16 bases average 3.9/2.8 points, but AIME drops 6.1/10.0. On a 16 GB machine where weights cannot fit, the prerouter yields +80-84% decode; on the 24 GB tier it adds only 0.5 tok/s. Limitations: single-request FIFO, a 44 ms/step CPU graph-building floor, gains shrinking on hot caches, MLX-only implementation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A per-layer prerouter can predict the next layer's routing one token ahead, and consuming the prediction as the routing itself makes the staged expert set identical to the routed set, letting SSD reads overlap compute for +80-84% decode when the checkpoint does not fit in RAM (§3.2, §5.3).
- Streaming experts from SSD bounds peak active memory to the active set — 2.9 GiB for a 19.5 GB 35B checkpoint at 20.4 tok/s, roughly 5× the fully-resident mlx-lm baseline on the same 24 GB machine (§3.1, §4, Table 1).
- An unmerged recovery LoRA trained on the student routing path recovers most of the joint int4-plus-routing-replacement loss (mean gaps 3.9/2.8 points over five benchmarks), whereas merging into int4 and re-quantizing destroys most of the adapter effect (18% logits-level retention) (§3.3, §5.2).
- The prerouter's advantage is conservation, not prediction quality: both arms read within a few percent of the same bytes per step at K=2/K=8, and the win equals the cold-read latency moved off the critical path (§5.3, Table 4).
Strengths
Prediction-as-routing eliminates the coverage/quality tradeoff by construction
Because the prerouter's prediction IS the decode-time routing (§3.2), staged slots map exactly to routed experts and no tokens are dropped or fallback-loaded; the approximation cost is moved into a three-phase training recipe (§4) instead of being paid at runtime, a cleaner formulation than Pre-gated MoE's runtime handling [17].
Measurement methodology is unusually candid and controlled
§5.1 uses same-session rotated A/B (arms in separate processes, order rotated, identical replayed token sequences, 3-round medians, matched cache budgets), openly reports ±40% run-to-run and 2.3× across-session spread, and refuses to use cross-session numbers as evidence — rare discipline for this hardware class.
Mechanistic accounting of where the win comes from
Concrete, quantified case for the unmerged LoRA
Engineering rigor and full artifact release
Weaknesses
No external offloading baseline measured
Prerouter contribution on the flagship machine is marginal; abstract blurs this
Quality results reported without variance, with one unexplained anomaly
Missing ablation separating the four stacked sources of quality loss
Single-request, FIFO-only serving
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Edge0 and what problem does it solve?
Edge0 is an open-source streaming inference engine that runs large Mixture-of-Experts (MoE) language models on a single consumer machine by keeping the expert weights on SSD instead of RAM. It targets the 'weights half' of the memory wall: a 35B-class MoE needs about 19.5 GB even at 4-bit, which crowds out a 24 GB desktop, while the activated experts per token are only ~3B parameters.
How does Edge0 reach 20 tokens/s serving a 35B MoE from an SSD?
Two mechanisms. First, experts are memory-mapped and streamed on demand, so RAM holds only the active set (~3 GiB). Second, a small per-layer 'prerouter' network predicts which experts the next layer will need one token in advance; the prediction is used as the actual routing, so the prefetched experts are exactly the ones consumed and the SSD reads overlap computation instead of stalling it. A recovery LoRA compensates for 4-bit quantization and the approximate routing.
Does Edge0 lose accuracy compared to the original fp16 model?
On average it stays close: across five OpenCompass benchmarks (AIME, HumanEval, GPQA-Diamond, MMLU-Pro, IFBench), the 35B and 8B tiers trail their fp16 bases by mean 3.9 and 2.8 points respectively. The loss concentrates in long-chain reasoning: AIME drops 6.1 points for the 35B tier and 10.0 for the 8B tier, while MMLU-Pro for the 8B tier is actually above its fp16 base.
Is the Edge0 code and the models publicly available?
The paper states the framework (github.com/Edge0-AI/Edge0), both model checkpoints, and the recovery-LoRA/prerouter adapters are released under Apache-2.0 on Hugging Face (Edge0-35B-A3B-preview and Edge0-8B-A1B-preview). This review did not independently verify the artifacts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as captions and inline descriptions; curve shapes and axis details were not analyzable, though the underlying numbers appear in Tables 1-4.
- The open-source artifacts (GitHub/Hugging Face links) are claimed in the paper but were not verified during this review.
- Base models (Qwen3.6-35B-A3B, Ling 3.0 tiny) are very recent 2026 releases whose specifications are taken from the paper alone.
- Some measurement details (exact prompts, seeds, KV-cache settings) are absent from the text, so a few protocol judgments rely on the authors' descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.