The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?
Srikanta Datta Tumkur, Srikanta Datta Tumkur, Jay Iyer, Jay Iyer, Mehar Simhadri, Mehar Simhadri, Sai Pavan Kumar, Sai Pavan Kumar, et al.
cs.AI
Quick verdict
Measures 54 Qwen2.5-7B anchors on 3 GPUs and simulates a cost-quality-latency atlas; candid and useful, but the frontier rests on unvalidated simulation.
Summary
The paper tackles the incomparability of LLM-inference speedup claims by placing quantization, KV-cache compression, speculative decoding, batching, and (simulated only) sparse attention on shared cost, quality, and latency axes. The authors measure 54 anchor configurations of Qwen2.5-7B-Instruct under vLLM 0.12 on RunPod L4, A100, and H100 GPUs (18 per GPU across three campaigns; total spend $18), with a GSM8K quality arm (200 questions, 5-shot, greedy) covering FP16, AWQ-4bit, FP8 weights, and FP8 KV cache. A Vidur-style profiled simulator calibrated against the anchors fills a 36-configuration grid on which Pareto frontiers and per-regime winners are computed; cross-campaign drift is below 1.5%. Key results: stacked combinations reach the frontier more often than single methods (9/15 vs 9/21, 60% vs 43%); FP8 weights retain 99.4% of FP16 accuracy at 0.61-0.65x latency and appear in three of four regime winners; AWQ-4bit reaches 0.34x per-token latency on L4 but loses 5.9% strict GSM8K accuracy, which flexible extraction shows is formatting-driven; a naive FP8 KV cache sustains normal throughput yet scores 0/200; n-gram speculation measures at parity (0.90-0.98x). Main limitations: one model and one measured task (GSM8K of four protocol workloads), interpolative simulator validation with no held-out points, a never-measured sparse-attention component inside the headline low-cost winner, and no stated code or data release.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A measure-then-simulate pipeline — 54 measured anchors plus a calibrated profiled simulator — can place the LLM inference optimization stack on common cost-quality-latency axes with cross-campaign drift below 1.5%.
- Stacked optimization combinations reach the Pareto frontier more often than single methods (9/15 vs 9/21, 60% vs 43%), and quantization appears in 13 of 18 frontier members.
- FP8 weight quantization is the quality-preserving workhorse: 99.4% of FP16 GSM8K accuracy at 0.61-0.65x baseline latency on all three GPUs, appearing in three of four regime winners.
- Quality-aware accounting overturns speed-only conclusions: a naive FP8 KV cache sustains normal throughput yet answers 0/200 GSM8K questions correctly, and AWQ-4bit's 5.9% strict accuracy loss is formatting-driven since flexible extraction ties FP16.
- The dominant configuration shifts with constraint and GPU (H100 wins tight latency at $4.62/Mtok; A100 wins throughput and cost at $0.106/Mtok), and as implementable in vLLM 0.12, n-gram speculative decoding adds nothing (0.90-0.98x).
Strengths
Joint accounting catches a trap speed-only benchmarks miss
The naive FP8 KV cache sustains normal measured throughput yet scores 0/200 on GSM8K (Sec. V-F) — a configuration a throughput benchmark would recommend. This is the paper's thesis validated by its own measurement, and Table III shows the quality guards excluding it from every regime.
Deployment-decision framing with re-costing built in
Table III names a dominant configuration per constraint with explicit quality and $/Mtok, hourly prices are stated (L4 $0.39, A100 $1.39, H100 $2.89), and the conclusion quantifies sensitivity: a 50% A100 price shock moves cost and quality-floor winners to the L4.
Unusually candid limitations section
Honest fidelity accounting separated from on-anchor fit
Actionable hardware-conditional regularity
Weaknesses
Evidence base far narrower than the atlas framing
Frontier rests on simulation with no held-out validation
The headline low-cost winner contains an unmeasured component
The 0/200 FP8-KV collapse is underexplained
Boundary statistical decisions encoded as verdicts; no code release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the best quantization for serving a 7B LLM like Qwen2.5-7B?
According to this paper, FP8 weight quantization is the safest default: it kept 99.4% of the FP16 model's GSM8K accuracy while cutting per-token latency to 0.61-0.65x baseline consistently across L4, A100, and H100 GPUs. AWQ 4-bit is faster on cheaper GPUs (down to 0.34x latency on the L4) but lost 5.9% strict GSM8K accuracy in this study, with the loss traced to answer formatting rather than arithmetic ability.
Does FP8 KV cache hurt model accuracy?
In this paper's measurement, a naive FP8 KV cache (without calibrated scales) in vLLM 0.12 sustained normal throughput but answered 0 of 200 GSM8K questions correctly — a total quality collapse. The authors note they tested only this naive path, and that scale-calibrated FP8-KV or eviction methods (H2O, PyramidKV) might avoid the collapse; those were left untested. The practical lesson: never adopt KV-cache quantization on throughput numbers alone.
Is speculative decoding worth enabling in vLLM?
For this configuration, no. The paper measured n-gram (prompt-lookup) speculative decoding on Qwen2.5-7B under two prompt designs and got 0.90-0.98x baseline latency — i.e., slightly slower than plain decoding. Draft-model speculation (EAGLE-3) could not be tested because vLLM 0.12's engine does not implement it and no EAGLE head exists for Qwen2.5-7B-Instruct, so the negative result applies only to the n-gram variant on a 512-token-prompt/128-token-output workload.
Which GPU gives the cheapest per-token cost for LLM inference?
In this study's July-2026 RunPod pricing (L4 $0.39/hr, A100 $1.39/hr, H100 $2.89/hr), the A100 80GB delivered the lowest cost at $0.106 per million tokens for the winning throughput/cost configuration, and also won the quality-floor-at-minimum-cost regime ($0.289/Mtok). The authors note these rankings are price-sensitive: a 50% A100 price shock would move the cost winners to the L4, so the atlas should be re-costed for current prices.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Central figures (Fig. 5 atlas, Fig. 2/6/7 frontiers, Fig. 10 fidelity plot) are referenced but not analyzable from the extracted text; all quantitative judgments rest on numbers quoted in prose and Tables I-III.
- The composition of the 36-configuration grid is not fully enumerated in the text, so grid-level judgments (e.g., which points are measured vs simulated) are inferences from the prose.
- Extraction is single-column running text with occasional merged headers, but it appears complete (abstract through reference [24]); no obvious truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.