Where Should a Document Live: Context, Representations, or Parameters?
Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias, Adrià de Gispert
cs.CL · cs.AI
Quick verdict
Controlled study finds KV-cache Cartridges beat parametric knowledge injection in accuracy and composability; plausible but model-dependent (Gemma replication fails).
Summary
The paper presents a controlled comparison of five ways to give LLMs access to new document knowledge: in-context learning (ICL/RAG), KV-cache methods (Cartridges, Compaction), and parametric adaptation (LoRA, MLP adapters, full fine-tuning). All methods train on identical LLM-generated self-study data under a KL-distillation objective and are evaluated on Qwen3-8B (plus Gemma-3-12B) across five knowledge-intensive benchmarks in single-document oracle and multi-document retrieval settings, with storage-matched adapter sizes. In the oracle setting, Compaction at 2x compression nearly matches ICL (73.4 vs 73.6 average) while Cartridges (70.6) lead all parametric methods (LoRA 64.5, MLP 62.2, full FT 59.8); Cartridges stay flat up to 100x compression where Compaction collapses. In the retrieval setting, Cartridges are the only method matching RAG, while merged parametric adapters collapse (e.g., LoRA on TechQA: 57.4 to 23.5 at k=3). Forgetting analysis shows only Cartridges, full fine-tuning, and large MLP adapters degrade general capabilities (~6% average, up to 16 points on HumanEval), arguing that full-rank updates, not parameter count, drive forgetting. Main limitations: only two mid-sized models, and the Gemma-3-12B replication contradicts the headline Cartridge results (37.3 average, severe instability), and no statistical significance testing.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In the oracle single-document setting, KV-cache Cartridges are the most accurate injection method at nearly every storage budget, outperforming parametric methods by ~10 points (Table 2: 70.6 vs 64.5 for the best parametric LoRA at the fixed size).
- Compaction matches the in-context oracle only at low compression (2x: 73.4 vs 73.6) and falls behind parametric methods above 50x compression.
- In the multi-document retrieval setting, Cartridges are the only composed method matching text RAG, while averaged parametric adapters and concatenated Compaction caches degrade monotonically with k.
- Only Cartridges, full fine-tuning, and large MLP adapters cause catastrophic forgetting (~6% average degradation, 13-16 points on coding), implying that the full-rank nature of an update, not its parameter budget, drives forgetting.
- Full fine-tuning is dominated on every axis: most expensive to train and store, least accurate, and most prone to forgetting.
Strengths
Genuinely controlled, storage-matched design
All five methods train on the same self-study corpus with the same KL-distillation objective (Section 4), and LoRA rank / MLP bottleneck are chosen to match the KV-cache memory footprint at each compression rate (2x-100x sweep, Section 4 Hyperparameters and Figure 1), isolating the representation-vs-parameters question.
Contamination confound explicitly addressed
Unlike prior QA-based comparisons on Wikipedia (Ovadia et al., Su et al.), the No Context baseline (24.9 average vs 73.6 for ICL, Table 2) confirms documents are genuinely novel, with the largest gap on FinQA (2.9 vs 66.8).
Composition failure localized to composition itself, not the merge operator
Forgetting tied to a mechanistic observation
Concrete runtime analysis, not just accuracy
Weaknesses
Second-model replication contradicts the headline claim
No statistical significance testing
Headline deltas are not traceable to any table
Full fine-tuning result may be an artifact of limited tuning
Single retrieval pipeline and LLM-judged metric without validation statistics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does it mean to 'inject' a document into an LLM?
Instead of pasting a document into the prompt for every query (in-context learning / RAG), knowledge-injection methods encode it once into either a compressed key-value cache prefix (representation-based, e.g., Cartridges or Compaction) or into fine-tuned weights (parametric, e.g., LoRA). At inference the model reads the encoded form instead of the raw text, which removes the per-query cost of re-processing the document.
Is fine-tuning better than RAG for adding new documents to an LLM?
In this paper's controlled comparison on five knowledge-intensive benchmarks with documents unseen in pretraining, no: per-document LoRA adapters averaged together collapsed in the multi-document retrieval setting (e.g., TechQA accuracy fell from 57.4 to 23.5 with just 3 retrieved documents), while the KV-cache Cartridge method matched text-based RAG. Parametric methods were cheapest per query but less accurate and not composable across documents.
What is a 'Cartridge' in LLM research?
A Cartridge (Eyuboglu et al., 2025) is a small trainable KV-cache prefix created via 'self-study': a generator LLM writes synthetic questions about a document, and the target model distills its own in-context answers into the cached vectors. At query time the cartridge replaces the document text. This paper finds Cartridges the most accurate and the only composable injection method, but they cost storage proportional to document length and cause mild catastrophic forgetting.
Does encoding documents into model weights hurt general abilities (catastrophic forgetting)?
It depends on the rank of the update, according to this paper: full fine-tuning, large full-rank MLP adapters, and Cartridges degraded control benchmarks (math, coding, instruction following, general knowledge) by about 6% on average (up to 16 points on HumanEval coding), while LoRA and the Compaction KV method stayed at base-model level. The authors argue the low-rank constraint itself, rather than parameter count, preserves general capability.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear only as raw axis-label text in the extraction; curve values cannot be verified, so figure-dependent claims (size sweeps, multi-document margins) rest on the numbers cited in body text.
- The extracted text shows OCR/layout noise (e.g., interleaved figure legends), and some appendix content (e.g., wider-adapter results in Appendix A) is described but not shown numerically.
- Review is based on arXiv v1 extracted text; the PDF rendering and any supplementary code were not accessible, and the paper states no code release.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.