A Universal Context-Reuse Layer for Cross-Model KV Sharing
Yi Li, Dongming Jiang, Yi Zhao, Bingzhe Li
cs.LG · cs.AI
Quick verdict
跨模型 KV 转换可在 3 对模型组合下，以接近基线的质量将目标 prefill 成本降低最高达 67%；虽然前景可观，但论文中并未具体说明转换器（translator）的具体结构。
Summary
这篇论文解决了当多个 LLM 服务于同一上下文时的冗余预填充（prefill）计算问题：即使另一个模型已经处理了相同的输入，每个模型通常仍会从零开始重建其 KV 缓存。作者提出了一种学习型的“传输（transport）”模块，将源模型的 KV 缓存转换到目标模型的表示空间，使冻结的目标模型能够直接从转换后的状态进行解码。文中评估了三种配置。在家族内（within-family）的 Qwen2.5-7B→1.5B 切换中，在 116 个 LongBench2 评分样本上，将冻结的 1.5B 模型的准确率从 27.59% 提升至 34.48%，弥补了其与原生 7B 模型（45.69%）差距的 38.1%。在跨家族（cross-family）的 Qwen2.5-1.5B→Gemma-2-2B 中，在 4K token 时增量预填充延迟最高降低了 67.04%，且在 1-512 的所有时间跨度（horizons）内，解码困惑度（perplexity）均接近原生基线。在 Llama3.1-70B→Qwen2.5-7B 中，模型保留了 96.3% 的原生目标准确率（44.0% 对比 45.7%），延迟从 899ms 降至 138ms。论文诚实地指出了切换（handoff）延迟排除了源模型的预填充时间。主要局限在于：转换器的架构、训练数据和损失函数从未明确说明；未发布代码；样本量小且缺乏显著性检验；此外还有一个未解释的困惑度异常，即切换模型在 H=1 时表现最好，但在 H=16 和 H=512 时却落后于原生模型。
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Translated KV states from Qwen2.5-7B lift a frozen Qwen2.5-1.5B target's LongBench2 accuracy from 27.59% to 34.48% (116 samples), recovering 38.1% of the gap to native 7B inference (45.69%).
- Cross-family Qwen2.5-1.5B→Gemma-2-2B handoff reduces incremental target-side prefill latency by up to 67.04% at 4K tokens while keeping decoding perplexity near native baselines across horizons 1–512.
- Cross-family Llama3.1-70B→Qwen2.5-7B handoff retains 96.3% of native target accuracy (44.0% vs 45.7%) while cutting measured incremental latency from 899ms to 138ms.
- KV caches can be treated as transferable computational assets ('context mobility') across model boundaries rather than strictly model-local artifacts.
Strengths
Well-chosen heterogeneity ladder
The three source→target pairs (§6.1) each isolate one axis — scale (Qwen2.5-7B→1.5B), family at matched scale (Qwen2.5-1.5B→Gemma-2-2B), and family+scale (Llama3.1-70B→Qwen2.5-7B) — making the difficulty progression interpretable rather than confounded.
Honest latency accounting conventions
§6.1 explicitly states handoff latencies measure only incremental reuse of an already-available source cache, and Table 4 decomposes the 34.5/53.8ms handoff into translation (9.1–12.2ms), peer copy, and assembly — openly leaving a 19.7–33.5ms remainder unattributed instead of hiding it.
Current, correctly-differentiated related work
Internally consistent headline numbers
Weaknesses
The central component is never specified
No comparison against the nearest prior translators
No uncertainty quantification anywhere
Unexplained anomalies, including one arithmetically wrong claim
Train/eval mismatch and minor inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes **cross-model KV sharing**: a learned translation layer T_θ that converts the KV cache produced by a source LLM into a representation consumable by a different target LLM, which then decodes without performing its own full prefill (§4–§5). Three source→target pairs span increasing heterogeneity: within-family Qwen2.5-7B→Qwen2.5-1.5B, comparable-scale cross-family Qwen2.5-1.5B→Gemma-2-2B, and large-to-small cross-family Llama3.1-70B→Qwen2.5-7B. Central claims: (1) within-family handoff lifts frozen-1.5B LongBench2 accuracy from 27.59% to 34.48% on 116 scored samples, recovering 38.1% of the gap to native 7B (45.69%, Table 1); (2) cross-family handoff cuts incremental target-prefill latency by up to 67.04% at 4K tokens (Table 5) with near-native decoding perplexity across horizons 1–512 (Table 6); (3) the 70B→7B handoff retains 96.3% of native target accuracy (44.0% vs 45.7%, Table 7) at 138ms vs 899ms. The related-work section (§2.2) is current and correctly positions the work against CacheBlend, KVCOMM, DroidSpeak, Heo et al.'s closed-form linear mapping [14], and Mixture-of-Translators [15].
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is cross-model KV cache sharing?
A serving technique in which the KV cache that one LLM computes while reading a prompt (the source) is converted by a learned translation module into the KV representation of a different LLM (the target). The target model then starts decoding from the translated state instead of recomputing the whole prompt, saving prefill time when models share the same context.
How much latency does KV handoff save in this paper?
Measured incremental handoff cost: 34.5ms vs 158.7ms target prefill for Qwen2.5-7B→1.5B at 8K–16K contexts (4.6× faster); up to 67.04% cheaper than native Gemma-2-2B prefill at 4K tokens; and 138ms vs 899ms for Llama3.1-70B→Qwen2.5-7B (6.5×). These figures exclude the source model's own prefill, which is assumed already computed and reused.
Does using another model's KV cache hurt accuracy?
Results are mixed. Within-family (Qwen 7B→1.5B), the frozen 1.5B model scored 34.48% on LongBench2 versus 27.59% with its own cache — an improvement. Cross-family Llama3.1-70B→Qwen2.5-7B kept 96.3% of native accuracy (44.0% vs 45.7%). Gemma-2-2B perplexity stayed near native levels — better at some decoding horizons, worse at others (H=16 and H=512).
Is code available for this paper?
No. The paper provides no repository link and never specifies the translation module's architecture, training data, or loss function, so the experiments cannot be reproduced from the text alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's visual content and the demo video are outside the text extraction (normal for PDF-to-text); all five tables and references [1]–[15] were captured in full, and the abstract is complete in the full text even though the metadata snippet cut mid-sentence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.