Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
cs.AI
Quick verdict
Sim study: KV tiering's 73× session gain is pure capacity, not policy; predicted-reuse and prefetch recommendations fail; honest but limited to batch-one sim.
Summary
This simulation study asks which placement policy should govern KV-cache blocks across GPU HBM, CPU DRAM, and SSD tiers for long-lived LLM sessions. Using a discrete-event simulator (Algorithm 1) with recency, reuse-frequency, the shipped predicted-reuse policy, and a new EWMA-predictive policy, plus prefetch lookahead sweeps, on synthetic chat, agent, and document-QA workloads, the authors deliver four findings. Tiering at a 1:8:64 capacity shape raises concurrent sessions per GPU by 73.02x and cuts cost per session 62.04x, but this tracks the tier capacity multiple (1+8+64=73), not placement quality. At batch one, decode is compute-bound (~13.24 ms/token, stall under 1% of TPOT), so policy moves PCIe migration traffic and TTFT rather than throughput: recency wins chat (27.38 GB vs 62.99 GB, 2.30x less traffic), reuse-frequency wins agent (49.36 GB) and document QA (48.23 GB). The shipped predicted-reuse policy is byte-identical to LRU; a genuine EWMA predictor differs but still loses to reuse-frequency where prediction was expected to win. Prefetch, even with a future-reading oracle, never beats no-prefetch across the grid on a contended link, and the cost-latency frontier is nearly vertical (8.38x cost span vs at most 10.1% P99 TTFT movement). Limitations: synthetic workloads favoring matched policies, batch-one-only regime, assumed fixed tier prices, and an abstract-mentioned random-forest calibration never evaluated in the body.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Tiering the KV cache across HBM/DRAM/SSD at a 1:8:64 shape yields 73.02x more concurrent sessions per GPU and 62.04x lower cost per session, entirely attributable to added capacity rather than placement policy.
- At batch one, placement policy does not move throughput; it controls PCIe migration traffic and TTFT, where recency wins chat (2.30x less traffic) and reuse-frequency wins agent and document QA (~1.6x less).
- The shipped predicted-reuse policy is byte-identical to recency, and a genuine EWMA predictor, while behaviorally distinct, still finishes second to reuse-frequency on the workloads prediction was meant to win.
- Prefetching does not pay on a contended link: even a future-reading oracle never beats no-prefetch in any cell of the policy x cache-size grid.
- The cost-latency frontier over tier compositions is nearly vertical: cost spans 8.38x while P99 TTFT moves at most 10.1%, so cheap SSD-heavy compositions dominate.
Strengths
Unusual negative-result honesty
The paper dismantles its own roadmap: the shipped predicted-reuse policy is shown byte-identical to recency (Sec. V-C, Table IV), the '+ prefetch' recommendation is refuted with an oracle upper bound (Sec. V-E), and the 73.02x headline is explicitly re-attributed from policy to tier capacity (Table III, Sec. V-A).
Clean mechanism-vs-policy separation
The ratio sweep (Fig. 6, Sec. V-A) shows tiered/HBM-only sessions-per-GPU flat at 72.68-73.22x across workloads and ratios, matching the 1+8+64=73 capacity multiple; this makes the capacity-accounting argument essentially airtight.
Oracle-bounded prefetch evaluation
Reproducibility hygiene
Deployment-useful frontier analysis
Weaknesses
Synthetic-workload circularity
Batch-one-only operating point
Unvalidated calibration
Numeric inconsistency and fixed price assumptions
Pivotal claims verified off-page
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper studies a policy question, not a systems build: for long-lived LLM sessions (multi-turn chat, agent loops, repeated document QA), which KV-block placement policy across GPU HBM, CPU DRAM, and SSD minimizes the cost that placement actually controls? The authors evaluate recency (LRU), reuse-frequency, the 'predicted-reuse' policy from their own roadmap, and a newly added EWMA next-touch predictor ('predictive'), plus prefetch lookahead arms, in a discrete-event simulator (Algorithm 1, Sec. III) over synthetic workloads (Table I). Five central claims emerge: (1) the 73.02x sessions/GPU and 62.04x cost-per-session wins of tiering (Table III) are capacity-accounting effects of the 1:8:64 tier shape, not policy effects; (2) at batch one, policy moves PCIe migration traffic and TTFT, not throughput (chat is 1307.60 tok/s under every policy); (3) recency wins chat (27.38 vs 62.99 GB, 2.30x), reuse-frequency wins agent (49.36 GB) and document QA (48.23 GB); (4) shipped predicted-reuse is byte-identical to LRU and the genuine EWMA predictor still loses where prediction was supposed to win (53.16 vs 49.36 GB on agent); (5) prefetch, even with a future-reading oracle, never beats no-prefetch on a contended link (Sec. V-E). The claims are unusually self-critical and mostly well supported within the paper's stated regime.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much does KV cache tiering improve LLM serving capacity?
In this simulation study, adding CPU DRAM and SSD tiers at a 1:8:64 capacity ratio raised concurrent sessions per GPU by about 73x and cut cost per session about 62x for long-lived chat, agent, and document-QA workloads. Importantly, the authors show this gain comes purely from the added storage capacity (1+8+64=73), not from smarter cache placement.
Which KV cache placement policy should I use for LLM serving?
Per this paper's simulator results at batch one: recency (LRU) minimizes PCIe migration traffic for concurrent multi-turn chat (2.30x less than frequency-based), while reuse-frequency wins for agent loops and repeated document QA (~1.6x less traffic). The paper also found its shipped 'predicted-reuse' policy was byte-identical to LRU, and a genuine EWMA predictor still lost to reuse-frequency.
Does prefetching the KV cache help performance?
Not in this paper's model. Under a contended-bandwidth link at batch one, even an oracle prefetcher that can see the future never beat doing no prefetch across the entire policy and cache-size grid, because speculative fetches delay demand fetches that are already waiting. The authors note the result may differ where spare bandwidth exists.
Is this paper based on real hardware experiments?
No. All results come from a discrete-event simulator of GPU HBM, CPU DRAM, and SSD tiers running synthetic chat, agent, and document-QA workloads. The authors acknowledge that on real traces the policy separation margins would compress, and that larger batch sizes would change the stall economics that drive several conclusions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions; per-cell values of Fig. 5/7/8 are partly quoted in prose but the plots themselves could not be analyzed.
- The random-forest calibration may be documented in an appendix not included in the extracted text; its absence from the body is judged as written.
- Table II is partial: block granularity and prefetch lookahead are listed only as 'swept' with no values, limiting exact reimplementation.
- Repository files (results/*.csv, cost_model.py) are referenced throughout but no public release URL appears in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.