SGD-KV: Summarization Guided KV Cache Compression
Zeyu Liu, Woomin Song, Xuandi Fu, Sai Muralidhar Jayanthi, Vivek Govindan, Aram Galstyan, Sravan Babu Bodapati, Srikanth Ronanki
cs.CL
Quick verdict
KV budget by summarization-head score cuts cache 75% with solid 1M-token results; train/eval overlap and no variance stats temper the SOTA claim.
Summary
This paper introduces SGD-KV, a head-aware KV cache compression method for long-context LLM inference. It first runs a chunk-summarization diagnostic: multiple short documents are concatenated, the model must identify the chunks and extract keywords, and each attention head is scored by how strongly attention flows from generated keywords back to their source text. Normalized scores then determine each head's KV budget (with sink/window reservations and water-filling redistribution), while token selection within a head reuses SnapKV-style cumulative attention. On Qwen2.5-7B-Instruct-1M — additionally fine-tuned because the original checkpoint performed poorly on MRCR — and Qwen3-32B at a 25% KV budget, SGD-KV reaches 34.16 on MRCR at 1M tokens versus HeadKV's 28.90, and 28.38 average on ETHIC (Qwen3-32B) versus 28.53 for full KV. Ablations are unusually thorough: reversing the allocation collapses accuracy even with 3x cache, and a generic summarization prompt recovers most query-aware performance. Main limitations: the fine-tuning data includes synthetic MRCR, GraphWalks and BABILong samples overlapping the evaluation tasks; all results are single runs without variance or significance testing; the AdaKV baseline behaves near-randomly, suggesting miscalibration; the SOTA claim is not uniform across lengths; and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A chunk-summarization diagnostic task systematically identifies attention heads specialized in hierarchical information aggregation, with stable rankings across datasets (IoU > 0.9).
- Allocating per-head KV cache budgets proportionally to summarization scores achieves state-of-the-art long-context accuracy at a 25% cache budget (75% reduction) on contexts up to 1M tokens.
- On MRCR at 1M tokens with fine-tuned Qwen2.5-7B-1M, SGD-KV reaches 34.16 accuracy versus HeadKV's 28.90 at the same budget.
- A fixed, task-agnostic summarization prompt serves as an effective proxy query for token selection when the real query is unavailable (Table 3).
- The score is causally informative: inverting the allocation collapses accuracy even with 3x cache (Table 5).
Strengths
Novel diagnostic grounds a new functional head class
The chunk-summarization task (Sec. 3.1-3.2) — concatenated documents, chunk counting, keyword extraction, and Eq. 1 attention scoring — is a genuinely new lens beyond retrieval-centric diagnostics; Appendix A.3 shows head rankings are stable (IoU > 0.9 within Dolly subsets) and generalize across six summarization datasets.
Strong negative controls validate the score causally
SGD-KV (Reverse) collapses to 8.21 at 128k despite 3x cache, and DuoAttention (Sum., Reverse) falls to 5.69-6.26 at long lengths (Table 5); the (thr) and Ipt. variants further map how low-score heads matter at ultra-long context.
Credible ultra-long-context results
Practically useful proxy-query finding
Unusually broad ablation coverage with honest omissions
Weaknesses
Train/eval overlap via fine-tuning data
No statistical treatment
Uneven baseline fidelity
SOTA framing exceeds the tables
Unexplained numeric inconsistency and unstated serving costs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This NeurIPS 2025 Efficient Reasoning workshop preprint (arXiv:2609.03235) tackles KV cache memory growth in million-token LLM inference. Three central claims: (1) a chunk-summarization diagnostic reliably identifies a functionally distinct class of 'summarization heads'; (2) allocating per-head KV budgets in proportion to summarization scores (with water-filling redistribution) achieves state-of-the-art accuracy at a 25% cache budget — a 75% memory reduction — on contexts up to 1M tokens; (3) a generic summarization prompt can substitute for the real query during token selection (Table 3). Models: Qwen2.5-7B-Instruct-1M and Qwen3-32B; benchmarks: MRCR, ETHIC, BABILong.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SGD-KV and how does it compress the KV cache?
SGD-KV is a KV cache compression method for long-context LLM inference. It scores every attention head with a chunk-summarization diagnostic task — the model reads concatenated documents, identifies the chunks, and extracts keywords — then allocates each head a KV cache budget proportional to its summarization score, while always keeping initial sink tokens and a recent-token window. Within each head, tokens are kept based on cumulative attention scores, as in SnapKV.
How much memory does SGD-KV save and at what accuracy cost?
At a 25% KV cache budget (a 75% memory reduction), the paper reports MRCR accuracy of 34.16 at 1M-token context on a fine-tuned Qwen2.5-7B-1M (full KV: 43.29), and near-parity with full KV on the ETHIC benchmark with Qwen3-32B (28.38 vs 28.53 average). Accuracy on shorter contexts and against some baselines is more mixed.
Is code for SGD-KV available?
The reviewed paper text does not mention any released code or repository. Reproducibility currently relies on the method equations (Eq. 1-2, Algorithm 1) and the hyperparameters documented in the appendices, and re-running 1M-context evaluations requires substantial GPU resources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2 and Figure 4 are described only in prose; their underlying numeric values (IoU curves, budget-sweep accuracies) are absent from the extracted text, so those results could not be checked numerically.
- Appendix A.7 Table 5 covers 8k-512k only; the authors state that 1M-context ablation results were omitted for compute reasons, so no judgment is made about that missing scale.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.