BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference
Anish Saxena, Jae Hyung Ju, Hritvik Taneja, Po-An Tsai, Aamer Jaleel, Christos Kozyrakis, Moinuddin Qureshi
cs.DC · cs.AR · cs.LG
Quick verdict
Claims wave-aware page placement adds HBM+host bandwidth for LLM decode (+4.3% TPOT, +31% tput, GH200); rigorous and honest, though single-platform evidence.
Summary
The paper argues that LLM decode wastes the host-memory bandwidth available on tightly coupled CPU-GPU systems (Grace Hopper), where coherent C2G links provide roughly 10% of HBM bandwidth. Current serving either ignores host memory or uses prefetching, which spends HBM write bandwidth and bounds effective demand bandwidth at (1−α). The authors define Concurrent and Proportional (CAP) access: every GPU wave must hit both tiers in proportion to their bandwidth ratio. Because 2MB pages make waves touch only tens of pages, decade-old proportional placement (BAPP-R) loses per-wave concurrency. BOOST is a ~800-line runtime extension of vLLM with two components: a Page Allocation Manager performing modulo-based page placement plus superpages for static weights, and a Data Shape Remapper that maps KV head chunks to tiers at runtime for attention. On GH200 (α=10%), iso-batch TPOT improves 4.3% on average while prefetching degrades 6%; throughput rises 31% on average in high-throughput serving, though the authors show most of that comes from added capacity, with only ~4% from bandwidth concurrency (Tables 3-4). Limitations: evaluation on one GH200 system plus an H100-based emulation for α=12% and tensor-parallel runs; synthetic workloads; dense models slow down at batch ≥256 because host loads bypass L2; no code release stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fully utilizing HBM+host bandwidth requires concurrent and proportional (CAP) accesses within every GPU wave; 2MB pages break the implicit per-wave concurrency of prior proportional placement, leaving BAPP-R 3.5% below HBM-only against a +10.4% ideal (§2.4-2.6, Fig. 4).
- Prefetching caps effective demand bandwidth at (1−α) because every migrated byte consumes an HBM write, degrading iso-batch TPOT by 6%, versus a (1+α) bound for direct concurrent demand loads (§2.5, §5.2).
- Wave-aware modulo page placement for static weights and runtime KV-head-chunk mapping for attention enable CAP without kernel changes, via ~800 LoC in vLLM (§3-4, §5.1).
- On GH200, BOOST improves iso-batch TPOT by 4.3% on average and high-throughput serving throughput by 31% on average — with most of the throughput gain from added capacity and only ~4% from bandwidth concurrency (§5.2, §5.5, Tables 3-4).
- Gains grow with the tier bandwidth ratio α, with projections approaching 3x on future α≈40% systems (§5.12, §6.1, Fig. 22).
Strengths
Crisp, quantified problem decomposition
The CAP framing (concurrency + proportionality, §2.4) is backed by measurements: prefetching bounds demand bandwidth at (1−α) with a 211GB/s demand loss (§2.5, Fig. 3), and BAPP-R lands 3.5% below HBM-only against a +10.4% ideal due to ±3.6% per-wave ratio variance on 2MB pages (§2.6, Fig. 4).
Honest gain attribution
§5.5/Table 3 uses a capacity-matched baseline to isolate the bandwidth-concurrency component at ~4% (344 vs 359 TPS), and Table 2 decomposes PAM (1.23x) from PAM+DSR (1.32x) — the authors refuse to let the capacity-driven 31% headline carry the novelty claim.
Real-system, kernel-transparent implementation
Candid failure-regime characterization
Validated forward-looking model
Weaknesses
Single-platform main results
Modest novel gain
Synthetic workloads only for main results
Thin statistical reporting
Baseline and setup caveats
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
BOOST (arXiv 2609.13592; Georgia Tech + NVIDIA Research) targets the decode phase of LLM inference on tightly coupled CPU-GPU systems such as Grace Hopper, where a cache-coherent C2G link gives host memory ~10% of HBM bandwidth (measured 350 vs 3330 GiB/s, α=10%, §5.1). The central claim is that extracting additive bandwidth (1+α) requires per-wave Concurrent and Proportional (CAP) accesses, and that both dominant alternatives fail: prefetching caps effective demand bandwidth at (1−α) because every migrated byte consumes an HBM write (§2.5, Fig. 3 shows a 211GB/s demand-bandwidth loss), while bandwidth-proportional page placement (BAPP-R, Agarwal et al. 2015) loses per-wave concurrency on modern 2MB pages — measured 3.5% below HBM-only against a +10.4% ideal (§2.6, Fig. 4). BOOST claims to be the first runtime providing CAP without kernel changes, via wave-aware modulo page placement for static weights and runtime KV head-chunk mapping for attention (§3-4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does BOOST solve?
On tightly coupled CPU-GPU servers like NVIDIA Grace Hopper, LLM inference either uses only HBM bandwidth or prefetches from host memory — the latter wastes host-memory bandwidth and consumes HBM write bandwidth. BOOST makes each wave of GPU threadblocks read from HBM and host memory simultaneously and in proportion to their bandwidth ratio (about 10% on GH200), extracting roughly 10% more memory bandwidth without modifying GPU kernels.
How much faster is BOOST?
On a Grace Hopper GH200 running a modified vLLM, BOOST improves time-per-output-token by 4.3% on average at fixed batch size (while prefetching degrades it by 6%), and improves throughput by 31% on average in high-throughput serving. The authors show most of that throughput gain comes from the extra host-memory capacity; only about 4% comes from concurrent bandwidth use, which is the genuinely novel part.
Does BOOST require changing model or GPU kernels?
No. It works purely through page placement (modulo-based allocation of model weights across HBM and host memory) and runtime KV-cache block management inside the serving system — about 800 lines of change to vLLM. Closed-source cuBLAS matmul kernels and FlashAttention-3 run unmodified.
When does BOOST not help?
Per the paper's own analysis: on PCIe or CXL attachments the bandwidth ratio is only ~2%, so gains are marginal; dense models slow down (up to 0.5%) at batch sizes above 128 on Hopper because host loads bypass the L2 cache; and MoE models lose about 2% at very small batches (16) where few experts activate. Recommended decode batches are 64–1K for MoE and 8–128 for dense models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Plotted data points in figures (e.g., per-configuration values in Figs. 13-18) are not present in the text extraction; the review relies on figure captions and aggregate statistics stated in the body text.
- Table 1's comparison grid is scrambled by extraction; its row/column content was reconstructed from the caption and surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.