OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
Zikun Li, Yixuan Mei, Shiqi Pan, Zixuan Chen, Xiaowen Zhang, Mengdi Wu, Shuhuai Lin, Yutong Yang, et al.
cs.DC · cs.AI
Quick verdict
OpWeave: flexible operator disaggregation for LLM serving; strong theory and real-H100 wins, but the headline 1.89× heterogeneous gain is simulation-only.
Summary
OpWeave is an end-to-end framework for operator-level disaggregated serving (ODS) of LLMs, splitting decode-phase computation into finer stages than the fixed attention–FFN boundary of prior AFD systems and mapping stages across heterogeneous GPU groups. It contributes (1) an analytical cost model proving that homogeneous ODS beats colocation only once KV-cache pressure pushes the feasible batch below the GEMM compute-bound threshold, with gain bounded by M/(M−Mweights), and that heterogeneous assignment gains are non-negative, unimodal in sequence length, and bounded by (1+√η)/2; (2) a regularity-aware planner that jointly searches operator partitions, hardware, parallelism, and batching over a partition-block abstraction using network-safe Pareto frontiers and branch-and-bound, exact within its disclosed search grid; and (3) a vLLM-based runtime (32K lines) with single native submission and GPU-resident decoding. On H100 clusters with Gemma-3-27B and Qwen3-Next-80B-A3B under TPOT SLOs, OpWeave is cheapest in 11 of 12 settings and up to 1.78× cheaper than the best feasible baseline, averaging 20.7 vs 124 pipeline stages, up to 22.8× less inter-node transfer than AFD, and over 80% occupancy. Simulated heterogeneous setups (H100+L40S, H100+A100) show up to 1.89× gains. Limitations: heterogeneous results are simulation-only (simulator validated on ≤8 GPUs), baselines are self-implemented special cases, no variance is reported, and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Homogeneous ODS beats colocated serving only when KV-cache pressure drives the feasible batch below the GEMM compute-bound threshold, with gain strictly increasing in sequence length and bounded by M/(M−Mweights) (Theorem 3.1).
- Heterogeneous operator assignment never costs more than the best homogeneous ODS, peaks at the homogeneous-cost crossing in sequence length, and is bounded by (1+√η)/2 across all cost ratios (Theorem 3.2).
- A regularity-aware planner over partition blocks synthesizes plans that are provably optimal within the configured finite search grid while keeping search tractable (Table 3 ablation confirms all variants find the same best plan).
- On real H100 clusters, OpWeave is cheapest in 11/12 model–context–SLO settings and up to 1.78× cheaper than the best feasible baseline, with 20.7 vs 124 average pipeline stages and up to 22.8× less inter-node transfer than AFD.
- In simulation on H100+L40S and H100+A100 clusters, heterogeneous OpWeave beats the best non-OpWeave baseline in all 12 settings by up to 1.89× and beats homogeneous OpWeave by up to 1.28×.
Strengths
Unified analytical characterization of when ODS helps
Theorems 3.1–3.2 (proofs in App. A.5) give closed-form conditions and tight bounds: homogeneous ODS gain activates only when KV pressure drops the relaxed batch b̄(s) below the GEMM threshold b* = Mweights·F/(2Pact·β), then grows monotonically capped by M/(M−Mweights); heterogeneous gain is non-negative, unimodal at the cost crossing x*, and ≤ (1+√η)/2 (instantiated as 1.10× for H100+L40S, 1.19× for H100+A100 in §7.4).
Planner exactness is empirically validated
Table 3 shows every ablated pruning configuration (Full, No-PF, No-B&B, Neither) finds the same best plan; Pareto-frontier pruning cuts simulator invocations 1.85×–3.75×, branch-and-bound cuts candidates ~46×, jointly speeding search 1.45× (H100) and 3.35× (A100+H100) — a rare direct check of an exactness claim.
Credible real-machine homogeneous results
Honest self-accounting throughout
Diagnosis of hybrid-attention imbalance
Weaknesses
Headline heterogeneous gains are simulation-only
Baselines are self-implemented special cases
Narrow workload and model coverage
No statistical rigor reported
No artifact availability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
OpWeave (arXiv 2609.14237v1, 13 Sep 2026; CMU + PKU) targets operator-level disaggregated serving (ODS): splitting decode-phase computation into stages finer than the fixed attention–FFN boundary of AFD systems (Step-3, MegaScale-Infer), and executing those stages across heterogeneous GPU groups. The paper structures its contribution around three claimed gaps: a theory gap (no unified account of when ODS reduces cost), closed by an analytical cost model (Theorems 3.1–3.2); a planning gap (fixed operator boundaries prevent joint search), closed by a partition-block planner; and a runtime gap, closed by a 32K-line vLLM-based execution layer. Headline numbers: up to 1.78× cheaper than the best feasible baseline on homogeneous H100 clusters, up to 1.89× on (simulated) heterogeneous clusters, meeting TPOT SLOs. Scope is steady-state decode only; prefill and arrival dynamics are out of scope by construction.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is operator-level disaggregated serving (ODS) in LLM inference?
ODS splits the decode phase of transformer inference into separately executed operator stages — for example, attention running on one group of GPUs and FFN/MoE on another — instead of executing whole layers together. Each stage can then be matched to hardware suited to its bottleneck (memory bandwidth for attention, compute throughput for GEMMs) and scaled independently.
When does disaggregating attention from FFN actually reduce serving cost?
Per the paper's Theorem 3.1, only after KV-cache growth pushes the colocated feasible batch size below the GEMM compute-bound threshold. Before that point, disaggregation gains nothing; afterward, the gain grows with sequence length but is capped by M/(M−Mweights) — essentially by how much GPU memory the model weights already occupy.
How much cheaper is OpWeave than prior disaggregated serving systems?
On real H100 clusters with Gemma-3-27B and Qwen3-Next-80B-A3B under latency SLOs, the paper reports OpWeave has the lowest cost in 11 of 12 tested settings and is up to 1.78× cheaper than the best feasible baseline among colocation, AFD, and CAD. In simulated heterogeneous clusters (H100+L40S and H100+A100), the reported advantage is up to 1.89× — but these heterogeneous numbers are simulation-only, validated on clusters of at most 8 GPUs.
Is OpWeave's code available?
The reviewed text contains no code or artifact release statement, and as of this review no public repository or project page could be found. The system is described as 32K lines of Python/C++ built on vLLM, and its planner depends on profiling tables that are not stated to be published.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were analyzed from extracted captions and embedded label values (e.g., Fig. 9–14 multipliers, Table 3 numbers); rendered plots, axes, and error bars were not visually inspectable.
- The task metadata lists 8 authors, but the paper text and arXiv listing both give 12 (adding Zhihao Zhang, Xupeng Miao, Rashmi Vinayak, Zhihao Jia); the review follows the full text.
- External verification (arXiv page, author homepage, GitHub search) confirms the paper is v1 preprint-only with no code release; venue status could change.
- Appendices A–D were included in the extraction, but any content existing only inside rendered figures may be incomplete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.