Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
cs.LG · cs.CL · cs.DC
Quick verdict
Fathom's per-query variable-depth key scans cut offloaded-KV decode 1.67x at 1M tokens; credible, unusually honest, but no HBM benefit and timing is batch-1.
Summary
Fathom targets sparse top-k decoding when the KV cache and its ranking index live in host memory, where the per-step scan of all n keys crosses PCIe and bounds decode time. Existing scans (Loki, Double Sparsity, SparQ) read a fixed bit depth per key; Fathom stores the 4-bit K cache channel-major as bit planes, so the first t planes of a channel are exactly its t-bit quantizer, and each query allocates its bit budget across channels by reverse water-filling over variance-weighted score importance. On an A100 with Qwen3-8B at one million tokens, a decode step is 1.67x faster in GPU time than the 136-bit scans and 2.50x faster than a landmark index; at matched GPU time with SparQ r=16 it reads 18% fewer bytes with lower attention error on six of seven settings, and it matches Double Sparsity's error at 46-74 bits. All per-token scans match the exact top-k oracle on RULER-style tasks; on 40 real coding-agent sessions Fathom reaches the strongest 136-bit scan's step agreement at 92 bits (32% fewer bytes). Limitations: no speedup when the index fits in HBM (arithmetic-bound kernel), offload timing is batch-1 and synthetic beyond 128k, fidelity uses one window per setting with no variance, agent runs stopped at 40 of 100 sessions, and end-to-end task success is unmeasured.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Storing the 4-bit K cache channel-major as bit planes makes any prefix of t planes exactly the channel's t-bit quantizer, so read depth can be chosen per query and per channel at no exactness cost.
- Per-query reverse water-filling over variance-weighted channel importance reaches Double Sparsity's error using 46-74 bits instead of 136 (a 1.8-2.9x byte saving) across seven settings up to 128k tokens.
- With KV rows and index in host memory, a Qwen3-8B decode step at 1M tokens is 1.67x faster in GPU time than the 136-bit scans, and at SparQ r=16's GPU time Fathom reads 18% fewer bytes with lower error on six of seven settings.
- On RULER-style tasks every per-token scan matches exact top-k decoding, and on real coding-agent sessions Fathom matches the best 136-bit scan's step agreement at 92 bits and reaches 0.67 agreement at 56 bits when k=2048, against SparQ r=16's 0.49.
- The speedup requires the index to live in a slower tier: with the index in HBM the scan is arithmetic-bound and Fathom is not faster.
Strengths
Genuinely new mechanism with an exactness guarantee
Channel-major bit planes make a t-plane prefix exactly the channel's t-bit mid-rise quantizer under one shared block scale (Sec 3.2), so partial reads are never approximations of approximations and form contiguous PCIe runs (Sec 3.6). Sec 3.3 derives the depth rule from the 4x-per-bit error reduction, and Appendix A's six-key worked example verifies the machinery end to end, including where it fails at B=6.
Unusual candor about where the method loses
The paper reports it is not faster with the index in HBM (Table 3 right, Sec 7), that wall-clock favors SparQ r=16 by 7% at 1M (Table 2), that the per-layer plan hurts on two of five models (A1), that Loki collapses on Qwen3 (B2), and that SparQ r=32 is more accurate on three of seven settings (Sec 5.2) — negative results most systems papers bury.
Matched-byte protocol with component-level attribution
Evaluation beyond synthetic proxies
Weaknesses
No variance on the fidelity numbers
Headline timing conditions are narrow
Benefit exists only in one hardware regime
Agent-session evidence is thin and proxy-level
Index memory cost and the unbuilt free path
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does the Fathom paper solve?
In LLM serving where KV caches are offloaded to host memory, each decode step must scan a compact index of all n keys across PCIe to pick the top-k for sparse attention, and at long context that scan's byte traffic bounds speed. Fathom reduces these bytes by letting each query read a variable number of bits per key channel instead of using a fixed-depth scan.
How does Fathom's bit-plane KV cache work?
Keys are stored as 4-bit uniform per-channel codes, packed channel-major so each bit position forms a contiguous plane. Reading only the first t planes of a channel yields exactly its t-bit quantizer under the same block scale, so a partial read is a true coarser quantization rather than a lossy approximation, and each query allocates its bit budget across channels by reverse water-filling.
How much faster is Fathom than SparQ, Loki, or Double Sparsity?
On an A100 with Qwen3-8B at one million tokens and the index in host memory, a decode step is 1.67x faster in GPU time than the 136-bit scans of Double Sparsity, Loki, and SparQ r=32, and 2.50x faster than a landmark index. Against SparQ r=16 at equal GPU time it reads 18% fewer bytes. The paper is explicit that with the index in GPU HBM it is not faster.
Does Fathom hurt answer quality?
On RULER-style retrieval tasks at 32k and 128k every per-token scan, including Fathom, stays within standard error of exact top-k decoding. On 40 real coding-agent sessions Fathom's generated steps agree with the exact top-k steps about as often as the strongest 136-bit baseline while reading about 32% fewer bytes; end-to-end task success was not measured.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-11 are available only as extracted captions and descriptions; plotted data (e.g., Fig. 3 error curves, Fig. 10 bandwidth) could not be checked against the tables.
- Appendix C is referenced but its narrative content appears absent from the extraction; Tables 12-23 are present though possibly reordered relative to the PDF.
- All baselines were re-implemented at 4-bit with the authors' channel-selection variants; fidelity of these re-implementations to the original systems could not be independently verified.
- Single-author arXiv preprint, not yet peer reviewed, and no code repository is stated anywhere in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.