Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Yi Xu, Ehsan K. Ardestani, Wenyin Fu, Martin Schatz, Krishna Malladi, Zhan Shu, Adnan Aziz, Shobhit Kanaujia, et al.
cs.DC · cs.AI · cs.LG
Quick verdict
Meta's Crossflow lends decode-node slack to prefill; well-ablated, honest work with 16-17% geomean gains, but private traces and no code limit verification.
Summary
The paper argues that static prefill–decode (P/D) disaggregation strands capacity because phase demand moves faster than replicas can be reassigned. Production-fleet telemetry shows uncached-input-to-output token ratios with peak-to-mean swings up to 4.7× at minute timescales, and the public TraceLab agentic trace shows a median 24.5× within-day hourly swing over 134 days, while replica reassignment takes tens of minutes on the authors' testbed. Crossflow keeps node roles fixed but makes the P/D boundary elastic: decode nodes publish short-lived, revocable multi-resource leases (compute, KV capacity, projected output, transfer, spill slots); a cluster scheduler performs shape- and cache-aware admission with atomic reservation and a pressure-scaled cluster-wide borrowing cap; and a decode-local executor advances bounded prefill units with per-unit revalidation, implemented in SGLang with two backends (chunked mixed-batch, PDMUX). Replaying one-hour segments of TraceLab and an internal trace at 8×–20× multipliers on six GB300 servers (2P4D), GPT-OSS-120B and GLM-5.2 gain 16.2–17.4% geometric-mean token throughput over static P/D and up to 43.4% at high load, with lower mean TTFT at every evaluated point; mean ITL regresses in some configurations (up to 34.3% for GLM-5.2 on TraceLab). Main limitations: internal traces and fleet data are unverifiable, no code release is stated, and evaluation covers only two MoE models on one GPU generation at six-server scale.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Phase demand in production LLM fleets varies far faster than static P/D repartitioning can follow: peak-to-mean ratio up to 4.7× at minute timescales and a median 24.5× within-day hourly range on TraceLab over 134 days.
- Sizing P/D pools at the 95th percentile strands 10.8–17.5% of a balanced configuration across four use cases, and asymmetric provisioning (prefill at median, decode at p95) reduces this to 6.0–7.4%.
- SLO-safe lending capacity on a decode node is nonlinear and request-shape dependent, so utilization-based heuristics mispredict it; placement alone changes SLO-safe prefill throughput by 11.3–34.0% at fixed aggregate decode load.
- Crossflow's lease-based elastic boundary improves token throughput 16.2–17.4% geometric mean over static P/D (up to 43.4% at high load) with lower mean TTFT at every evaluated point.
- Nine servers of static P/D still trail six-server Crossflow (−2.7%/−8.3% input/output throughput), so elastic lending outperforms adding hardware.
Strengths
Motivation grounded in production-scale and public evidence
§3.1 combines fleet telemetry (minute-scale ratio CV 0.14–0.34, peak-to-mean 4.7×) with 134 days / 2,959 hours of the public TraceLab trace (median within-day hourly range 24.5×, median CV 0.89), and quantifies stranding: 95th-percentile sizing strands 17.5/10.8/13.8/12.0% across four use cases.
Honest reporting of adverse results
§6.2 reports mean ITL regressions (GLM-5.2/TraceLab +20.9–34.3%, GPT-OSS/internal +2.5–6.7%); §6.3 reports a losing configuration (4P2D: input −2.8%, TTFT +33.3%) and internal-trace p99 ITL up to +5.0%. Rare candor for a systems paper.
Unusually thorough ablations that attribute the gain
Hardware-equivalence framing answers the operator question
Backend-independent executor interface, implemented twice
Weaknesses
Half the evidence is proprietary and unauditable
No artifact release statement
Controller constants are unspecified
Narrow evaluation surface vs fleet-scale motivation
Replay methodology is a synthetic load model
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Crossflow (Meta; arXiv 2609.27085) targets the static capacity boundary of prefill–decode (P/D) disaggregated LLM serving. The authors first characterize the problem with production-fleet telemetry and the public TraceLab coding-agent trace: the uncached-input-to-output token ratio moves with peak-to-mean ratios up to 4.7× at minute timescales (Figs. 1–2), the hourly ratio swings with a median 24.5× range within a day across 134 days (§3.1), and reassigning a replica costs 8:35–17:31 on their testbed — so pools sized at the 95th percentile strand 17.5/10.8/13.8/12.0% of a balanced configuration across four use cases. The system claim is that decode nodes can lend bounded, SLO-safe compute to prefill without changing roles: each decode node publishes a short-lived, revocable multi-resource lease (compute, KV, projected output, transfer, spill slots); a cluster controller performs shape- and cache-aware, two-sided admission with atomic reservation and a pressure-scaled cluster-wide concurrency cap; and a decode-local executor advances prefill in bounded units with per-unit revalidation. Headline result: 16.2–17.4% geometric-mean token-throughput improvement over static P/D across two models and two traces, up to 43.4% at high load, with lower mean TTFT at every evaluated point.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does Crossflow solve in LLM serving?
In prefill-decode disaggregated serving, the GPU split between prefill and decode pools is fixed, but the actual ratio of input to output token demand fluctuates constantly. Crossflow lets decode nodes temporarily lend bounded compute and memory capacity to prefill work without changing node roles, recovering the capacity that static partitioning strands.
How much faster is Crossflow than static P/D disaggregation?
On GPT-OSS-120B and GLM-5.2, replaying a public agentic trace and an internal trace on six GB300 servers, token throughput improves 16.2–17.4% on geometric mean and up to 43.4% at high load, with lower mean time-to-first-token at every tested load. Decode inter-token latency worsens in some configurations, by up to 34.3% for GLM-5.2.
Is Crossflow's code publicly available?
The paper describes an implementation in SGLang (a ~6.7K-line Python shim plus ~2,807 lines across 15 SGLang files) but does not state that the code, configurations, or internal traces are publicly released.
What is a lease in Crossflow?
Each decode node periodically publishes a short-lived, versioned, revocable lease stating how much compute, KV-cache capacity, projected output work, transfer bandwidth, and spill slots it can safely lend to prefill. The cluster scheduler must atomically reserve these credits before redirecting a request, and the node revalidates them before executing each bounded unit of work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as garbled embedded text; values cited from Fig 3/5/6/8/9 annotations could not be visually verified against axes and labels.
- The internal evaluation trace and all production-fleet data (§3) are proprietary; claims resting on them cannot be independently checked.
- Controller constants (pressure thresholds, lease TTLs, G(Q̂P) shape, score weights) are described qualitatively, not tabulated, in the extracted text.
- Metadata lists 8 authors; the full text adds Ajit Mathews and Chunqiang Tang (Meta Platforms).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.