Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
cs.SE · cs.DC · cs.LG
Quick verdict
Rigorous paired study: prefix caching breaks LLM-serving reproducibility, amplified by weight quantization; convincing controlled evidence, but small scale.
Summary
This empirical study measures how prefix caching — reusing key/value tensors for shared prompt prefixes across requests — undermines reproducibility in LLM serving, and how weight quantization amplifies the effect. Using a paired design (greedy decoding, fixed seed, batch size one, serial requests), the author ran an 80-episode multi-turn agentic tool-use workload (BFCL) and a GSM8K single-turn bridge twice per arm on llama.cpp and vLLM across four weight formats (F16 to 3-bit). With caching disabled, all 800 repeated episodes were bit-identical, bounding other nondeterminism below 0.5%. With caching enabled, cross-arm trajectory divergence rose from 36.2% (F16) to 75.0% (Q4_K_M) for Qwen2.5-7B on llama.cpp; a Cochran–Armitage test confirms the trend (p=1.3e-8). Three controlled experiments attribute run-to-run divergence to llama.cpp's server-level prompt cache (disabling it cuts divergence from 38.8% to 1.2%), and a reset control shows both paths reproduce given cache state (40/40 each) while differing from each other on 14/40. On GSM8K, correctness flips occur in both directions with no aggregate shift (McNemar p=0.26). Limitations: only 7–14B open-weight models, one consumer GPU, two English benchmarks, agentic success rates near the floor, and some single-run cells.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- With prefix caching disabled under greedy decoding, fixed seed, and batch size one, repeated execution is bit-identical (0/800 episodes across ten configurations), bounding other nondeterminism at ≤0.5%.
- Enabling the cache changes agent trajectories on 36.2% of episodes at 16-bit and 75.0% at 4-bit (Qwen2.5-7B, llama.cpp), and coarser weight quantification amplifies this divergence (Cochran–Armitage p=1.3e-8).
- Cached serving is deterministic given cache state: run-to-run divergence stems from server-side cache state (one llama.cpp flag moves it by 37.5pp), and both paths reproduce 40/40 under restored state while still differing from each other.
- Cache-induced divergence reaches task outcomes without directional bias: GSM8K correctness flips occur in both directions (13 vs 7) with no aggregate accuracy shift (exact McNemar p=0.26).
Strengths
Bit-identical control arm makes the attribution clean
With caching disabled, all 800 repeated episodes (10 configurations × 80) reproduced exactly, bounding every other nondeterminism source at ≤0.5% under greedy decoding, seed 42, batch size one (Section IV-A) — any later divergence is attributable to the cache by construction.
A confound was discovered mid-study and isolated rather than hidden
llama.cpp's server-level prompt cache (introduced Oct 2025) inflated within-arm divergence; the flag experiment shows 38.8% → 1.2% when disabled (37.5pp), the order effect (Table 1: 38.8% → 77.5% with an interposed pass) vanishes with the layer off, and both original and controlled measurements are reported side by side (Sections IV-B/IV-C).
The tautology objection is answered by experiment
Statistical care above venue norms
Honest error reporting increases trust
Weaknesses
Narrow external validity
The primary workload cannot answer the outcome question
Table 2's within-arm column mixes two cache layers
Cross-engine magnitudes are confounded
The amplification mechanism is inferential
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.04748, single independent author) makes an empirical claim about production LLM serving: prefix caching — reuse of key/value tensors for shared prompt prefixes — silently breaks output reproducibility, and weight quantization amplifies the effect. Five research questions structure the work: baseline determinism (RQ1), cache-induced divergence (RQ2), quantization interaction (RQ3), backend generality (RQ4), and task outcomes (RQ5). The headline numbers: 0/800 bit-identical episodes with the cache off; cross-arm trajectory divergence of 36.2% → 61.3% → 75.0% → 77.5% from F16 through Q3_K_M for Qwen2.5-7B on llama.cpp; a 37.5-percentage-point swing in run-to-run divergence from a single llama.cpp flag; and GSM8K correctness flips in both directions with no aggregate shift (McNemar p=0.26). The conceptual contribution is the reframing: cached serving is deterministic *given cache state*, and irreproducible in practice because that state is invisible to the request, absent from the response, and never reset by default.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does prefix caching change LLM outputs?
Yes. This study finds that reusing cached key/value tensors changes the floating-point accumulation path enough to flip sampled tokens: on BFCL agentic episodes, enabling the cache changed the trajectory on 36.2% of episodes at 16-bit precision and 75.0% at 4-bit (Qwen2.5-7B on llama.cpp), while aggregate accuracy on GSM8K was unchanged.
Is vLLM or llama.cpp deterministic with temperature 0?
In this study, with prefix caching disabled, greedy decoding at batch size one reproduced bit-identically in 800 of 800 episodes across two engines and four weight formats. With caching enabled, output became a function of the server's cache history; disabling llama.cpp's server-level prompt cache cut run-to-run divergence from 38.8% to 1.2% of episodes.
Does quantization make LLM inference less reproducible?
The paper reports that coarser weight quantization amplifies cache-induced divergence: cross-arm trajectory differences rose from 36.2% (F16) to 61.3% (Q8_0), 75.0% (Q4_K_M) and 77.5% (Q3_K_M) for Qwen2.5-7B on llama.cpp, a trend confirmed at p=1.3e-8 and saturating below 4 bits.
Should I disable prefix caching on my inference server?
The author argues no: aggregate accuracy was unaffected (GSM8K flips favored the cached path 13 times vs 7 for recompute, p=0.26). For workflows needing repeatability, the paper recommends pinning cache configuration, resetting cache state at run boundaries, and logging per-request cache exposure rather than abandoning caching and its throughput benefits.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is complete: the full abstract, all three tables (values inline), both figures' data labels, references [1]–[37], the data-availability statement and the author bio are present; no truncation detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.