MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
Tiancheng Zhang, Yulin Chen, Yunfeng Zhao, Shaoyuan Huang, Cheng Zhang, Xiaofei Wang
cs.AI
Quick verdict
Claims 42.6% mean/84.8% tail latency cuts for disaggregated LLM serving via device prediction + conformal calibration; promising, but only 2 decoders tested.
Summary
MAPS targets load imbalance in prefill-decode disaggregated LLM serving, where unknown output lengths cause decode instances to diverge in KV-cache pressure under round-robin routing. The framework adds three components: (1) a device-assisted speculative predictor (LoRA-tuned Vicuna-7B or 160M) that estimates output length on the request-origin device, overlapped with cloud-side prefill to hide latency; (2) uncertainty-aware calibration using one-sided conformal prediction over a sliding window of 500 requests, converting point estimates into upper bounds with target coverage (alpha=0.1); (3) hierarchical scheduling that routes requests to memory-feasible decoders with minimum queue length and applies SJF reordering within each decoder, with RR fallback on prediction timeout. Implemented on vLLM v0.13.0 with Mooncake KV transfer across one prefiller and two A6000 decoders, it is evaluated on ShareGPT (Poisson arrivals) and BurstGPT traces with LLaMA3.1-8B and Qwen2.5-3B. Reported results: mean end-to-end latency reduced 42.6% and P99 tail latency up to 84.8% versus vLLM, SGLang, and Llumnix; SP-7B reaches MAE 56 tokens on Alpaca; calibration cuts underestimation probability from 64% to 9%. Ablations attribute gains to each component, the 160M variant still beats all baselines, and MAPS approaches an Oracle using ground-truth lengths. Key limitations: a two-decoder testbed with deliberately heterogeneous memory caps, no larger models, chain-of-thought workloads out of scope, and no statistical uncertainty reporting.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Device-assisted speculative prediction runs on the request-origin device overlapped with cloud-side prefill, so length estimation adds negligible latency to the critical path.
- One-sided conformal calibration converts point predictions into output-length upper bounds satisfying P(Lout ≤ Lup) ≥ 1−α, cutting underestimation probability from 64% to 9%.
- Hierarchical global-local scheduling (memory-feasible minimum-queue routing plus intra-decoder SJF reordering) reduces mean E2E latency by 42.6% and P99 tail latency by up to 84.8% versus vLLM, SGLang, and Llumnix.
- MAPS is robust to predictor quality: the SP-160M variant still outperforms all three baselines, and full MAPS approaches an Oracle scheduler using ground-truth output lengths.
Strengths
Principled treatment of the failure mode that matters
Section 3.2 uses a one-sided nonconformity score (Eq. 2: positive only on underestimation) rather than generic interval prediction, matching the admission-safety need; Table 2 shows underestimation probability falling from 64% (SP) to 9% (SP+UAC) while a naive +10% offset only reaches 56%.
Overhead claims are measured, not asserted
Appendix D.1 quantifies both paths: Fig. 12 shows device-side prediction (tens of ms for SP-160M, several hundred ms for SP-7B on Jetson Orin NX) fully overlapped by cloud prefill, and Fig. 13 reports MAS dispatch at 1.26–1.59 ms (mean±sd over 1000 requests) versus thousands of ms of decode latency.
Unusually complete ablation and robustness matrix
Motivating phenomenon is quantified, not anecdotal
Honest, specific limitations section
Weaknesses
Testbed too small to support the global-scheduling claim
Memory heterogeneity is engineered into the experiment
Prediction quality evaluated only in-distribution
No statistical uncertainty on headline numbers
Device-assisted premise is idealized
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MAPS (arXiv 2609.15359) is a systems method paper on scheduling for prefill-decode (PD) disaggregated LLM serving. The motivating observation (Fig. 1a-b, Appendix A) is that output lengths are unknown at dispatch, so request-count-fair routing (round-robin) lets KV-cache pressure diverge across identically provisioned decoders — one decoder at ρ≈1.05 spends ~80% of runtime above a 0.85 utilization threshold while peers idle. The paper makes three claims: (1) a device-side LoRA-tuned predictor estimates output length hidden under cloud prefill at negligible cost (§3.1, Appendix D.1); (2) one-sided conformal calibration turns point estimates into bounds satisfying P(Lout ≤ Lup) ≥ 1−α (§3.2, Eq. 2–5); (3) hierarchical global-local scheduling — minimum-queue routing among memory-feasible decoders plus intra-decoder SJF — cuts mean E2E latency 42.6% and P99 up to 84.8% versus vLLM, SGLang, and Llumnix (§4.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MAPS and what problem does it solve?
MAPS is a scheduling framework for prefill-decode disaggregated LLM serving. Because a request's output length is unknown at arrival, round-robin routing can push requests onto decode servers whose KV cache is nearly full, causing queues, preemptions, and tail-latency spikes. MAPS predicts the output length on the user's device, converts the prediction into a statistically guaranteed upper bound using conformal calibration, and routes each request to a decoder with enough free KV memory and the shortest queue.
How much faster is MAPS than vLLM or SGLang?
On the authors' testbed (one prefiller and two NVIDIA A6000 decoders, LLaMA3.1-8B and Qwen2.5-3B, ShareGPT Poisson and BurstGPT bursty workloads), MAPS reduced average end-to-end latency by 42.6% and P99 tail latency by up to 84.8% compared with vLLM, SGLang, and Llumnix. These are single-run numbers on a small cluster, so treat them as best-case for this configuration.
Does MAPS require running a model on the user's device?
Yes. MAPS runs a LoRA-tuned lightweight LLM (Vicuna-7B, or Vicuna-160M for constrained devices) on the request-origin device to predict output length while the cloud performs prefill; the prediction is hidden under prefill latency. If the calibrated bound does not arrive within one second, the scheduler falls back to plain round-robin routing.
Is MAPS code or data publicly available?
The paper does not state a code release. However, it specifies all major ingredients: vLLM v0.13.0, SGLang v0.5.5, Llumnix as baselines, Mooncake for KV transfer, and public datasets (ShareGPT, BurstGPT, Alpaca), plus hyperparameters (α=0.1, 500-request calibration window, LoRA rank 16), so a faithful reimplementation is possible though laborious.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as axis-label text dumps; exact curve shapes and per-rate orderings are taken from prose statements, not from the plots themselves.
- Some extracted numbers (e.g., BurstGPT 'temporal scale' axis values 185–205, Fig. 9 annotation values 16%/80%/86%) may be extraction artifacts; figures could not be visually verified.
- Baseline versions are as reported (vLLM v0.13.0, SGLang v0.5.5); no independent verification of configuration fairness is possible from text alone.
- The paper header references ICML 2026 proceedings and an arXiv Sep 2026 stamp; peer-review status of the venue version is unknown.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.