Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Summary

The paper introduces SLOWeave, an online scheduler for LLM serving that adaptively chooses the prefill chunk size at each iteration: it admits the largest pending prefill chunk whose mixed prefill-decode iteration is predicted to finish before the earliest next-token deadline of active decode requests. The decision uses a monotone iteration-cost model (a profiled lookup table in the GPU runtime) searched in logarithmic time, with a prediction margin for safety. Under an accurate monotone cost model, the authors prove the choice is deadline-safe and maximizes immediate prefill progress (Proposition 1). Evaluation combines a dependency-free Python event-driven simulator with an iteration-level GPU runtime (8xA100 and 8xH100 nodes; 8B and 70B-TP8 models) across chat, mixed, long-context, and bursty workloads at TPOT targets of 10/25/50 ms. At 25 ms, SLOWeave improves goodput over the best fixed-chunk baseline by 39% on mixed and 38% on long-context traffic; at 10 ms the gains are 3.3x and 2.4x. GPU runs also show higher goodput and lower energy per valid request than the runtime default and fixed chunking. Limitations: Proposition 1 is essentially a direct consequence of the argmax definition; the Sarathi-style tuned baseline promised in Section 5.2 is not reported in Tables 2-3; no variance or confidence intervals are given for GPU measurements; and real cost non-monotonicities (kernel transitions, tensor-parallel communication, prefix-cache hits) are acknowledged but only partially addressed via P99 tables.