Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Zhenhe Wu, Yaping Jin, Qinghua Xing, Hang Zhou, Wei He, Xianjie Wu, Xianfu Cheng, Jian Yang, et al.
cs.CL
Quick verdict
Proposes joint MoE router-cache post-training; Qwen3 cache gains are large and well-ablated, but evidence is simulated traffic, not end-to-end latency.
Summary
The paper formulates expert-cache management in memory-constrained MoE inference as a model-side learning problem: post-training jointly adapts the MoE backbone and lightweight auxiliary routers while preserving the native Top-K inference rule. A Temporal Router learns same-layer expert retention after each access with no proactive loading; the full Spatio-Temporal mode adds a Spatio Router that uses the causal predecessor's hidden state to refine the cache before the target layer is accessed. Both modes are evaluated on Qwen3-30B-A3B and GPT-OSS-20B across GSM8K, MATH, and CommonsenseQA. The update-only Temporal Router beats LRU/LFU/LRFU on matched LM-only backbones in hit rate and traffic across all six backbone-task pairs. On Qwen3, the full mode improves adjusted hit rate by 1.15-18.03 points and cuts traffic by 4.6-53.3% versus ProMoE, the strongest evaluated prefetching baseline; GPT-OSS results are task-dependent. Ablations show joint post-training reaches far higher coverage than auxiliary-only training, and sweeps of cache capacity, refinement budget, and cache-loss weight expose an accuracy-coverage-traffic trade-off (full-mode accuracy drops 0.56-3.28 points on Qwen3; up to 10.84 points at loss weight 2.0). Main limitations: results are simulated decode-stage traffic rather than end-to-end latency, and five-seed means are reported without dispersion or significance tests.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cache-aware post-training can jointly adapt MoE expert demand and cache priorities while preserving the native Top-K inference rule.
- The update-only Temporal Router consistently improves hit rate and reduces traffic versus LRU/LFU/LRFU on matched LM-only backbones across all six backbone-task pairs, with zero proactive loading.
- On Qwen3, the full Spatio-Temporal mode improves adjusted hit rate by 1.15-18.03 points and reduces traffic by 4.6-53.3% versus ProMoE, the strongest evaluated prefetching baseline.
- Joint post-training achieves substantially higher cache coverage than auxiliary-only training (e.g., 73.13% vs 63.01% hit on Qwen3/GSM8K), at a measurable accuracy cost for the full mode.
- Cache capacity, refinement budget, and cache-loss weight sweeps expose a quality-coverage-traffic trade-off rather than a single optimal configuration.
Strengths
Proactive-load-aware, gaming-resistant metrics
Section 4.1 charges every proactive insertion as a full expert transfer: AdjHit = (A-D)/(A+P) and Load/token = (D+P)Sexp/T. This exposes that Temporally Extended MoE's 100% raw hit comes with restricted routing, worse traffic (3299 vs 974 MB/token on Qwen3/GSM8K) and lower accuracy (81.50 vs 85.44).
Matched, well-documented baselines
LRU/LFU/LRFU run on matched LM-only post-trained backbones (same data, optimizer, schedule), and four learned prefetchers (Least-Stale/SpecMD, ProMoE, FineMoE, Temporally Extended MoE) are configured with explicit settings in Section 4.1 (e.g., FineMoE: 1K-entry map store, lookahead 3, 64 candidates).
Deep ablation coverage
Honest scope discipline
Deployment-friendly design details
Weaknesses
No dispersion or significance reporting
Headline gains are Qwen3-specific and quality-costed
Simulated traffic, no system comparison
Confounded comparison via model reshaping
No code or checkpoint release; heavy compute
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets decode-stage expert-weight transfer in memory-constrained MoE serving and reframes cache management as a model-side learning problem: post-train the backbone jointly with auxiliary routers so expert demand and cache priorities co-adapt, while the native Top-K rule stays intact. Two deployable modes are offered — update-only Temporal Router (retention, no proactive loads) and full Spatio-Temporal Router (adds causal pre-access refinement over top-R candidates). Headline claims (Abstract, Table 1): Temporal Router wins hit rate and traffic over LRU/LFU/LRFU on all six backbone-task pairs; on Qwen3 the full mode beats ProMoE by 1.15-18.03 adjusted-hit points and 4.6-53.3% traffic. Claims are mostly well-scoped — the abstract flags GPT-OSS task dependence — but omit the full mode's accuracy cost (0.56-3.28 points on Qwen3) and the GPT-OSS/GSM8K traffic regression (2951 vs 2109 MB/token vs ProMoE).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does this paper demonstrate faster MoE inference?
No. It measures simulated decode-stage expert-weight traffic (MB per decode token) in a trace-driven simulator, not wall-clock latency or throughput. The authors state that realized speedup depends on scheduling and transfer-computation overlap, which are not evaluated (Appendix D).
Does the cache-aware training hurt model accuracy?
The update-only Temporal Router largely preserves accuracy (e.g., 85.44% on Qwen3/GSM8K, unchanged from baseline). The full Spatio-Temporal mode drops accuracy by 0.56-3.28 points across Qwen3 tasks, and strong cache-loss weights cost more: 10.84 points at weight 2.0.
Which models and benchmarks were used?
Qwen3-30B-A3B-Instruct-2507 (48 MoE layers, 128 experts per layer, top-8 routing) and GPT-OSS-20B (24 layers, 32 experts, top-4), evaluated on GSM8K, MATH, and CommonsenseQA (the validation split, since test labels are not public).
How is this different from expert prefetching methods like ProMoE?
Prefetching methods predict expert demand on a fixed model. This paper post-trains the model jointly with lightweight auxiliary cache routers so expert demand itself becomes more cache-friendly, while its adjusted metrics still charge every proactive load. On Qwen3 it beats ProMoE on adjusted hit rate and traffic; on GPT-OSS results are task-dependent.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2 heatmap cells and Figure 3 curve values are only available at the aggregate endpoints quoted in the text (e.g., effective experts 53.41 to 34.54); per-layer figure data cannot be verified from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.