Disaggregated Quantization: Specializing LLM Prefill and Decode
Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi, Tijmen Blankevoort, Dan Alistarh
cs.LG
Quick verdict
Specializes LLM prefill/decode quantization with separate weights/formats; strong systems results, yet single-run training and batch-1 latency evals limit it.
Summary
The paper proposes disaggregated quantization (DQ): choosing quantization formats, weights, and storage placement separately for LLM prefill (compute-bound) and decode (memory-bound) phases. Its training tool, QADD, uses the SFT label mask to route prompt tokens through a prefill pathway and response tokens through a decode pathway within one KL-distillation pass against a frozen BF16 teacher. Three schemes are evaluated on Qwen 3 (0.6–8B) and Gemma 3 (1–12B): format disaggregation (NVFP4 prefill compute with weight-only decode), improving decode-heavy accuracy by 1.8–4.5 points at no cost; full disaggregation (separately trained prefill weights), gaining up to 10.7 points at 2-bit decode while speeding prefill 1.49×; and offloaded disaggregated prefill (ODP), which streams prefill weights from SSD for a 1.78× time-to-first-token speedup at 8K context in llama.cpp on DGX Spark. For released Qwen3.8-27B GGUF decoders, trained NVFP4 prefillers more than double 1-bit accuracy (+32.5 MMLU-Pro, +35.3 MMMU-Pro), though 3-bit decode slightly degrades. PTQ-only format disaggregation on models up to 2.8T parameters yields significant gains in 6 of 13 model–benchmark pairs. Limitations: single training runs (error bars cover checkpoint variation only), batch-one latency only, no MoE or multi-turn evaluation, and single 27B evaluations.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Disabling activation quantization only at decode (format disaggregation) improves decode-heavy accuracy by 1.8–4.5 family-mean points with no storage or speed penalty (§3.2, Table 1).
- Training separate NVFP4 prefill weights (full disaggregation) improves low-bit decode accuracy — up to +10.7 points for LUT2 decode-heavy on Qwen 3 — while accelerating prefill 1.49× (Table 1).
- NVFP4 prefillers trained for frozen Qwen3.8-27B GGUF decoders raise IQ1_S accuracy from 29.04 to 61.54 on MMLU-Pro (+32.5) and 24.39 to 59.65 on MMMU-Pro (+35.3) without modifying the decode checkpoint (Table 5).
- ODP streams prefill weights from SSD with no added device memory, giving a 1.78× TTFT speedup at 8K context in llama.cpp on DGX Spark (Fig. 1, §3.2).
- PTQ-based format disaggregation on SOTA models up to 2.8T parameters yields significant gains in 6 of 13 model–benchmark combinations with no significant degradations (Table 2).
Strengths
Unusually complete evaluation matrix
Table 1 spans seven configurations × two model families with both cost (prefill/decode speedup, device GB) and accuracy on two workload types: decode-heavy (GSM8K, MATH-500, MMLU-Pro with reasoning on/off) and prefill-heavy (RULER 13 tasks × 4K–32K), all under real disaggregated vLLM serving.
Honest cost and confound accounting
Table 1 reports the storage doubling of full disaggregation (5.53→9.44 GB for Qwen3-8B LUT3) vs ODP holding 5.53 GB at 1.47× prefill; Appendix B.5 (Table 8) shows the +35.3 IQ1_S MMMU-Pro gain coincides with 54.8% shorter mean generations, ruling out the length confound.
Strongest statistics where it matters most
Deep ablations beyond the headline
Deployed-systems artifacts
Weaknesses
Single-run training with narrow error bars
Latency evidence limited to batch one on one platform
27B prefiller results are single evaluations
1-bit baseline is near-collapsed
ODP operating envelope is narrow
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces disaggregated quantization (DQ): choosing quantization formats, weights, and storage placement separately for prefill (compute-bound) and decode (memory-bound) inference. Three progressively stronger schemes are proposed: (a) format disaggregation (NVFP4 prefill compute + weight-only decode, e.g., NVFP4A16), (b) full disaggregation (separately trained prefill weights), and (c) offloaded disaggregated prefill (ODP, SSD streaming). The enabling tool, QADD (§2.1, Fig. 3, App. A.1), repurposes the SFT label mask to route prompt tokens through the prefill pathway and response tokens through decode within one KL-distillation pass. Claims: format disaggregation improves decode-heavy accuracy for free (§3.2); full disaggregation boosts low-bit decode accuracy while accelerating prefill (Table 1); 27B prefillers more than double 1-bit accuracy (Table 5); ODP yields 1.78× TTFT at 8K (Fig. 1); PTQ-only disaggregation scales to 2.8T parameters (Table 2). Claims are honestly scoped to single-user, dense models.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is disaggregated quantization in this paper?
It is the idea that LLM prefill (prompt processing, compute-bound) and decode (token generation, memory-bound) should use different quantization formats, weights, or storage placement. A training method called QADD jointly optimizes both pathways toward the same response objective using the SFT label mask to route tokens.
How much does the NVFP4 prefiller improve 1-bit GGUF models?
For released Qwen3.8-27B GGUF decoders, training an NVFP4 prefiller raised IQ1_S accuracy from 29.04% to 61.54% on MMLU-Pro (+32.5 points) and from 24.39% to 59.65% on MMMU-Pro (+35.3 points), leaving the decode checkpoint untouched. Gains shrink at 2-bit (~+6–7 points) and turn slightly negative at 3-bit.
What is ODP and when is it faster?
Offloaded disaggregated prefill streams the extra prefill weights from SSD through device buffers, avoiding additional GPU memory. On DGX Spark in llama.cpp it is faster than a resident weight-only baseline from about 4K context, reaching a 1.78× time-to-first-token speedup at 8K; at shorter prompts SSD loading makes it slower.
Does disaggregated quantization work on MoE models?
Not evaluated. The authors argue the SSD-streaming principle (ODP) does not transfer well to mixture-of-experts because fewer active parameters make loading far costlier than compute; ODP is presented mainly for dense local deployment, and no MoE experiments appear in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-level results are reconstructed from captions and embedded axis labels; individual training curves (Figs. 7, 10–15) cannot be verified from the extraction.
- Minor OCR risk on decimal values in dense tables (Tables 1, 5, 8, 10, 11); cited numbers match the abstract and body text where cross-checkable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.