Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
Atta Ul Asad, Ahsan Bilal, Muhammad Ali, Muhammad Haseeb, Dean F. Hougen
cs.CL
Quick verdict
Claims INT4 KV-cache quantization silently degrades RAG faithfulness despite preserved accuracy; well-controlled audit, but one 7B model and small data.
Summary
The paper asks whether offline KV-cache quantization — precomputing and compressing caches of retrieved documents — damages RAG faithfulness even when answer accuracy survives. Using Qwen2.5-7B-Instruct, the authors compare an oracle run (C0), a BF16 cache round-trip baseline (C1), INT8 per-token (C2), and INT4 group-wise (C3, groups of 64) on RGB and HotpotQA (300 examples each, K∈{1,3,5} retrieved chunks; 3,600 generations per dataset). A single causal prefill builds a unified cache, avoiding positional-stitching confounds, and a Stage-0 gate confirms C1 reproduces C0 exactly on 50 held-out examples. Faithfulness is measured with HHEM-2.1-Open hallucination rate, DeBERTa-v3-large NLI entailment, and a Claude Haiku 4.5 LLM judge (92% agreement with 50 human labels on RGB, 76% on HotpotQA). INT8 is near-lossless. INT4 lowers EM/F1 (correct-to-wrong flips 133 vs 10 on RGB) and, on the accuracy-preserved subset, faithfulness flips are overwhelmingly negative: 231 vs 24 on RGB (p<10^-38), 173 vs 31 on HotpotQA (p<10^-30), i.e., over 90% negative. Harm grows directionally with retrieval depth (slope p=0.12, underpowered) and with distractor fraction on RGB (β=0.22). INT4 also under-refuses on HotpotQA (0.513 to 0.317 at K=1) and degenerates on up to 6% of outputs. Limitations: one model family, 300-example benchmarks, a proprietary judge, and no released code.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 4
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- INT8 offline KV-cache quantization is near-lossless for both accuracy and faithfulness in RAG with Qwen2.5-7B-Instruct.
- INT4 quantization degrades faithfulness even on examples whose containment-EM outcome is unchanged, with over 90% of faithfulness flips negative (RGB 231:24, p<10^-38; HotpotQA 173:31, p<10^-30).
- Standard EM/F1 accuracy metrics are blind to this faithfulness regression, so accuracy-only evaluation systematically misses it.
- The INT4 harm grows with retrieval noise (distractor fraction, β=0.22 on RGB, significant at K=3 and K=5) and, directionally, with retrieval depth (slope p=0.12, underpowered).
- INT4 uniquely causes degenerate generations (up to 6% on RGB at K=5) and breaks refusal calibration on HotpotQA (0.513 to 0.317 at K=1), inflating surface metrics.
Strengths
Precisely identified and genuinely novel gap
Section 2 triangulates three literatures — KV compression (KIVI, KVQuant, ZipCache, KVTuner: accuracy/perplexity only), RAG faithfulness (RAGChecker, RAGTruth, HELMET: no compression), and offline-cache systems (TurboRAG, CacheBlend: no faithfulness) — establishing that no prior work audits faithfulness under offline cache quantization.
Confound-controlled experimental design
Section 3 builds one unified cache via a single causal prefill over system prompt + top-K chunks, explicitly removing the positional-stitching artifact of independent chunk caches; the Stage-0 gate (A.1) requires C1 (BF16 round-trip) to exactly reproduce C0 (oracle) on 50 held-out examples, so later deltas are attributable to bit-width alone.
Multi-signal faithfulness measurement with honest signal diagnostics
Self-identified refusal-calibration confound
Deployment-relevant, honestly accounted engineering numbers
Weaknesses
Single model at a single scale
Primary signal is a proprietary judge with thin calibration
H2 is underpowered by design choice
Small evaluation sets and lenient accuracy metric
No code, prompts, or quantization implementation details released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a narrow but well-chosen question: does compressing offline (precomputed) KV caches of retrieved documents degrade RAG faithfulness — whether responses stay grounded in retrieved evidence — independently of answer accuracy? It claims three novelties: (1) the first faithfulness audit of offline KV-cache quantization; (2) INT8 near-lossless versus INT4 silently harmful on accuracy-preserved examples (>90% of faithfulness flips negative; McNemar p<10^-20); (3) amplification by retrieval noise and depth. The gap analysis (§2) is precise and current: KV-compression work (KIVI, KVQuant, ZipCache, KVTuner) evaluates accuracy/perplexity only; RAG-faithfulness work (RAGChecker, RAGTruth, HELMET) ignores cache compression; offline-cache systems (TurboRAG, CacheBlend) ignore faithfulness. The motivating distinction — a correct answer string that is no longer supported by the given context — is real and well argued.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does INT8 KV-cache quantization hurt RAG quality?
In this paper's experiments with Qwen2.5-7B-Instruct on RGB and HotpotQA, INT8 was near-lossless for both answer accuracy (containment EM, token F1) and faithfulness measured by three independent signals. The harmful effects — accuracy loss, faithfulness degradation, degenerate outputs — were specific to INT4.
What is the difference between faithfulness and accuracy in RAG?
Accuracy compares the generated answer string to a reference answer (e.g., exact match, token F1). Faithfulness asks whether the answer is actually supported by the retrieved documents given to the model. A model can output the correct string without being grounded in the provided context — accuracy metrics cannot detect this, which is the paper's central motivation.
How much storage does INT4 KV-cache quantization actually save?
The paper measures about 3.6x reduction versus a BF16 cache at retrieval depth K=5, below the nominal 4x, because per-group scale and zero-point metadata consumes roughly 11% of the INT4 footprint. INT8 achieves about 1.9x versus the nominal 2x, with about 5% overhead.
Which model, datasets, and evaluation tools were used?
Qwen2.5-7B-Instruct, evaluated on RGB (300 examples) and the HotpotQA distractor split (300 examples) at retrieval depths K of 1, 3, and 5, with FAISS retrieval over bge-small-en-v1.5 embeddings. Faithfulness was scored with HHEM-2.1-Open, DeBERTa-v3-large NLI entailment, and a Claude Haiku 4.5 LLM judge; accuracy used containment exact match and token F1.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are images; their plotted per-condition metric levels (e.g., absolute hallucination rates per K) are not present in the extracted text — only the gaps in A.5, flip counts in §5, and caption annotations were available for numeric review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.