Dissecting GPU Utilization for LLM Inference on Nvidia Hopper
Mohammad Siavashi, Gerald Q. Maguire, Dejan Kostic, Marco Chiesa
cs.PF · cs.AR · cs.DC · cs.LG
Quick verdict
Claims Hopper decode GEMM underutilization stems from an m64 GMMA fragment floor, shown via 8 counter-pinned NCU views; rigorous but single-GPU in scope.
Summary
The paper argues that a single SM-utilization percentage hides the real state of LLM inference on NVIDIA Hopper, because it collapses occupancy, spatial coverage, stall behavior, and useful tensor work into one elapsed-cycle proxy. The authors profile vLLM with FlashAttention-3 and cuBLASLt on a single H100 NVL (BF16, TP=1) across four production models — Llama-3-8B, Qwen3-14B, Qwen3-32B, and the MoE Qwen3-30B-A3B — sweeping cold prefill, warm prefill (≥99% synthetic prefix-cache hit), and decode over sequence length and batch size, yielding 104 cells and 759 per-role rows. They replace the single number with eight NCU-counter-pinned views (Tab. 2). Key results: decode dense-GEMM SM-busy is 6.5–12.7% versus 73–97% in cold prefill; Hopper's GMMA m64 fragment floor caps decode fragment fill at 1.6–12.5% (B=1–8), so per-kernel compute SOL overstates useful matmul throughput by 8–64×; occupancy is co-limited by registers and shared memory at 12 warps/SM; a 204-entry cuBLASLt tile enumeration finds no 132-multiple grid for decode-QKV; and a B=32 validation arm confirms the compute–memory crossover near the predicted B*≈31 on Qwen3-14B (SM-busy ~12%→71.9%, 10.10→2.38 ms/token). Limitations: one GPU/SKU, BF16 only, TP=1, synthetic warm-cache states, an analytical launch-overhead constant, and no reported run-to-run variance.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single SM-utilization number conflates distinct denominators (elapsed-cycle pipe throughput, occupancy, spatial coverage, useful fragment fill) and systematically misreads decode behavior in LLM inference.
- Hopper's BF16 GMMA m64 fragment floor caps small-batch decode dense-GEMM fragment fill at 1.6–12.5% for B=1–8, so per-kernel compute SOL overstates useful matmul throughput by 8–64× — a measurement-attribution gap, not recoverable headroom.
- Decode dense-GEMM occupancy is co-limited by registers and shared memory at 12 warps/SM, and no cuBLASLt tile in the enumerated 59-pair space yields a 132-multiple CTA grid for decode-QKV.
- The compute–memory crossover batch B* for dense decode GEMMs lies at roughly 25–49; the B=32 validation arm confirms the crossing on Qwen3-14B and shows KV capacity, not parameter count, determines whether it is reachable.
- FA3 decode diverges from dense GEMMs in stall signature (stall_wait/barrier vs long_scoreboard, §5.10), implying attention needs kernel-side rework while the dense stream needs batching/memory levers.
Strengths
Counter-pinned methodology
Every one of the eight views (Tab. 2) is tied to a named NCU counter or explicit formula; §2.4's two-denominator occupancy treatment (FA3 main: 7.6 warps/SM = 12% of the 64-warp architectural ceiling but 95% of its 8-warp resource cap) resolves a real practitioner confound.
Falsifiable negative result via exhaustive enumeration
§5.7/Fig. 6 sweeps the full 204-entry cuBLASLt space (17 algoIds × 12 tiles) for Llama-3-8B decode-QKV: 59 producible (algoId, tile) pairs, zero reach a 132-multiple grid, with a structural explanation (132 = 4·3·11 vs N = 6144 = 2^11·3 excludes power-of-two tiles).
Self-validation arm
Cross-counter triangulation and attribution honesty
Regime-resolved mechanism separation
Weaknesses
No variance or repetition protocol
Wave-efficiency model assumes one CTA per SM
Synthetic endpoint for warm prefill
Analytical launch overhead never measured
Artifact only promised
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement/characterization study of LLM inference on a single NVIDIA H100 NVL (94 GB HBM3, BF16, TP=1), profiling the stock vLLM + FlashAttention-3 + cuBLASLt stack across four production models (Llama-3-8B, Qwen3-14B, Qwen3-32B, and MoE Qwen3-30B-A3B with top-8/128 routing), three regimes (cold prefill, warm prefill at a synthetic ≥99% prefix-cache hit, decode), and swept (S, B) — 104 cells, 759 per-role rows (Tab. 4). The central claim is that `sm__throughput.avg.pct_of_peak_sustained_elapsed` conflates at least four denominators, and that eight counter-pinned views (Tab. 2) recover the mechanism behind any utilization reading. Headline claims: (1) decode dense-GEMM fragment fill is capped at 1.6–12.5% (B=1–8) by Hopper's fixed m64 GMMA fragment, so per-kernel compute SOL overstates useful matmul throughput by 8–64× (§5.6) — explicitly framed as a measurement-attribution gap, not recoverable headroom; (2) decode/warm stall budgets are long_scoreboard-dominated (70–84%) while cold prefill shifts to barrier/gmma, with math_pipe_throttle absent from the top-3 of all 360 dense-GEMM rows (§5.4); (3) occupancy is register+SMEM co-limited at 12 warps/SM (§5.3, Tab. 5); (4) a 204-entry cuBLASLt tile enumeration finds no 132-multiple grid for Llama-3-8B decode-QKV (§5.7, Fig. 6); (5) a B=32 validation arm confirms the crossover near predicted B*≈31 on Qwen3-14B (dense SM-busy ~12%→71.9%, 10.10→2.38 ms/token) (§5.10, Tab. 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why is GPU 'SM utilization' misleading for LLM decode?
Decode GEMMs have tiny row counts (M equals batch size). Hopper's BF16 GMMA instructions always issue fixed 64-row fragments, so at batch 1 only 1/64 of each fragment's rows carry real token work. The SM-throughput counter charges the padded multiply as activity: the paper's example shows a 9.5% per-SM compute-SOL reading that corresponds to only about 0.11% of the H100's 835 TFLOP/s peak in useful matmul.
What hardware and models were profiled in this study?
A single NVIDIA H100 NVL (94 GB HBM3) running vLLM in BF16 with tensor parallelism 1, FlashAttention-3 and cuBLASLt, profiled with Nsight Compute 2024.3. Four models — Llama-3-8B, Qwen3-14B, Qwen3-32B, and the sparse MoE Qwen3-30B-A3B (top-8 of 128 experts) — across cold prefill, warm prefill, and decode: 104 configuration cells and 759 per-role rows.
How can small-batch LLM decoding be made more efficient, according to the paper?
Not by retuning cuBLASLt tiles — the m64 fragment floor and the missing 132-multiple grid are structural. The levers are raising the effective row dimension M (speculative, tree, or multi-token decoding; cross-request token packing), batching toward the measured compute–memory crossover at roughly B=25–49 within latency SLOs, and amortizing the ~30% launch overhead with persistent kernels or CUDA graphs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are text-extracted as label/value streams; every number cited in this review appears in the text, but some figure-only visual detail (e.g., exact per-reason bar heights in Fig. 4 at cells not quoted in prose) is not recoverable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.