PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving
Omkar Shewale, Deepak Kumar, Deepak Kumar, Divakar Kumar Yadav
cs.PF · cs.LG
Quick verdict
Solid single-H100 study: prefix-cache parity across vLLM/TRT-LLM with scheduler-driven TTFT flips; credible artifact but thin statistics.
Summary
The paper presents PrefixBench-H100, a benchmark characterizing prefix (KV-cache) reuse in two production LLM serving runtimes — vLLM 0.21.0 and TensorRT-LLM 1.2.1 (PyTorch backend) — on a single H100 NVL, using Qwen2-7B-Instruct in BF16 driven through identical OpenAI-compatible JSONL workloads. Synthetic repeated-prefix and RAG-template traces sweep prefix length, suffix diversity, concurrency, arrival pattern, output length, and cache configuration, collecting TTFT, throughput, and cache-hit statistics. Reuse-on yields 5–6.5× p50 TTFT reductions at 8192-token prefixes (Table 1), with both runtimes statistically indistinguishable in cache effectiveness (~92% hit). The central result is a scheduling-layer leadership flip: TensorRT-LLM wins p50 TTFT by 2.4× under burst at concurrency 32 (§4.2), while vLLM wins p95 TTFT by 9.2× under fixed-rate 16-QPS RAG load (§4.6); throughput favors vLLM by 2.8× (§4.12). tokens_per_block tuning shows no detectable effect (§4.4). Cache-pressure experiments (§4.8) show eviction onset tracks the working-set-to-KV-capacity ratio, validated on a 32B model and a capacity-capped 7B setup. Energy savings of 24–31% per request are reported (§4.10). Limitations include fixed-rate rather than Poisson arrivals, a scheduling account uninstrumented on the TensorRT-LLM side, single-GPU/7B focus, and thin statistical replication beyond one stability-checked cell.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On a 94 GB H100 NVL, prefix reuse cuts p50 TTFT by 5–6.5× at 8192-token shared prefixes for 7B-class models, with vLLM and TensorRT-LLM statistically identical in cache effectiveness (~92% hit).
- Cross-runtime TTFT leadership flips with arrival pattern — TensorRT-LLM leads p50 by 2.4× under burst at concurrency 32, vLLM leads p95 by 9.2× under fixed-rate 16 QPS — and the difference originates in the scheduling layer, not the cache.
- The documented tokens_per_block reuse-granularity tradeoff is not detectable in the TensorRT-LLM 1.2.1 PyTorch backend on H100 (§4.4 null result).
- Cache eviction onset is governed by the ratio of prefix working set to KV capacity rather than model size, with hit-rate parity between runtimes preserved under pressure (§4.8, Tables 3–4).
- Prefix reuse reduces per-request energy by 24–31% for long-prefix sequential workloads (§4.10).
Strengths
Matched-workload controlled design
Both runtimes are driven by one ~300-line aiohttp driver over identical JSONL workloads via /v1/completions (§3.3–3.4), and Table 1's reuse-on/off anchor converts the headline claim into a measured causal delta (345.1→53.0 ms on vLLM; 306.1→58.2 ms on TRT-LLM at pl=8192).
Rigorous cache-pressure isolation
Table 3 compares each cell to its analytical no-eviction ceiling (1 − np/N) rather than raw hit rates, and Table 4's capacity-capped 7B rerun (68.2%→3.6% hit on both runtimes) decouples KV capacity from model choice — the cleanest experiment in the paper.
Candor about causal limits
Strong public artifact
Two independent robustness axes
Weaknesses
Statistical replication limited to one cell
Internal inconsistencies
Promised ITL never reported
Half-instrumented causal mechanism
Narrow hardware/model envelope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PrefixBench-H100 (arXiv 2609.19657, cs.PF) is a measurement/characterization study, not a new algorithm. It asks where the bottleneck for prefix reuse sits on Hopper-class hardware now that KV memory is abundant (94 GB H100 NVL). Central claims: (1) reuse-on delivers 5–6.5× p50 TTFT reduction at 8192-token prefixes for 7B-class models (Table 1); (2) vLLM 0.21 and TensorRT-LLM 1.2.1 are statistically identical in cache effectiveness (~92% hit at 8192; §4.1, §4.3); (3) cross-runtime TTFT leadership flips with arrival pattern — TRT-LLM by 2.4× on p50 under burst at c=32 (§4.2), vLLM by 9.2× on p95 at fixed-rate 16 QPS (§4.6) — and the difference lives in the scheduler above the cache; (4) eviction onset tracks the working-set/KV-capacity ratio, not model size (§4.8); (5) tokens_per_block is a dead knob in the PyTorch backend (§4.4); (6) 24–31% per-request energy savings (§4.10).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much faster is TTFT with prefix caching on an H100?
For 7B-class models in this study, enabling prefix reuse cut p50 time-to-first-token by 5–6.5× at an 8192-token shared prefix (vLLM: 345 ms → 53 ms), with both vLLM and TensorRT-LLM achieving ~92% cache hit rates.
Which is faster on H100, vLLM or TensorRT-LLM?
It depends on the arrival pattern. Under bursty load at concurrency 32, TensorRT-LLM had 2.4× lower p50 TTFT; under steady fixed-rate RAG load at 16 QPS, vLLM had 9.2× lower p95 TTFT and 2.8× higher throughput. The paper attributes the difference to scheduling, not the cache.
When does prefix caching stop helping?
When the shared-prefix working set exceeds the GPU's KV-cache capacity. Degradation is driven by that ratio, not model size: at 13.2× over-subscription, hit rates fell to ~5.6% and TTFT regressed by roughly 15×.
Is the benchmark reproducible?
Per the paper, yes: the Zenodo artifact includes workload generators, 102 raw result files, and a notebook reproducing every figure without a GPU; a full re-measurement takes about 3 GPU-hours on a single H100 NVL.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as captions and axis text in the extraction; visual claims (CDF plateaus in Figure 5, Figure 7 heatmap cells) were taken from captions, not the plots.
- The extraction interleaves figure-axis labels into body text (e.g., inside §3.5 and §4.1), which may have garbled some sweep definitions.
- The Zenodo artifact (DOI 10.5281/zenodo.21725505) was not independently inspected for this review.
- Reported software versions (vLLM 0.21.0, TRT-LLM 1.2.1, CUDA 13.0, driver 580.105.08) could not be verified against any release channel.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.