Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir Uddin
cs.CL · cs.AI
Quick verdict
Honest laptop-scale pilot (1M-param models, 768 tokens): training objective, not architecture, creates the sink; the million-token question stays open.
Summary
This paper asks whether gated attention and hybrid linear-attention designs (the Kimi K3 pattern) actually eliminate attention sinks at million-token context. Its empirical core is SinkProbe, a four-metric suite (sink mass, massive activation, attention entropy, position-resolved recall with a signed recency gap), applied to four ~1M-parameter models trained identically except for token mixing: softmax, softmax+gate, 3:1 linear/global hybrid, and hybrid with attention residuals, all evaluated at 96-768 tokens. The strongest result is a control showing the training objective, not architecture, creates the sink: removing the auxiliary next-token loss cut sink mass from 0.316 to 0.049 with recall unchanged. Gating did not reduce sink mass because the gate never learned sparsity (mean score 0.686 vs 0.116 published), and both gating and attention residuals halved or cut massive activations while sink mass stayed within a seed spread (0.089-0.455) wider than the architectural effects. The hybrid stacks never learned retrieval (recall below the 8.3% chance floor), so they carry no position-bias evidence. An analytic cache model argues the 69/93 linear-layer mix makes 1M tokens affordable (~24 GiB vs ~2.5 TiB dense). The million-token question itself is left to a pre-registered protocol for released weights; experiments never exceed 768 tokens. Notably honest about its negative results and scale limits.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The training objective, not the architecture, produces the attention sink: removing the per-position auxiliary loss cut sink mass from 0.316 to 0.049 with recall unchanged (Table VII).
- Sink mass, massive activations, and position bias are dissociable: gating and attention residuals sharply reduce activations while sink mass stays within seed noise, and the stack with lower sink mass had the steeper position preference (Sections VI.F–VI.G).
- Gating did not reduce sink mass at 1M-parameter/1,500-step scale because the gate never learned sparsity (mean gate score 0.686 vs 0.116 published after trillions of tokens).
- The Kimi K3 layer mix (69 of 93 layers with non-growing cache) makes a 1M-token window affordable at roughly 24 GiB versus about 2.5 TiB for a dense stack, given two declared assumptions.
- The title question cannot be answered with laptop-scale proxies and requires executing the pre-registered protocol (Section VII) on released frontier weights.
Strengths
Causal control isolates the sink's origin
Table VII: the identical softmax stack shows 0.316 sink mass (worst layer 0.633) with a per-position auxiliary loss vs 0.049 (0.067) with answer-only loss; recall unchanged (32.0 vs 33.3%) and seed spread collapses from {0.455, 0.089, 0.405} to {0.051, 0.046, 0.050} — a single-variable effect far above noise.
Pre-registration discipline
Pass/fail thresholds (Table IV) are derived from published values before runs; Section VII fixes seven measurement lengths, sample sizes from Eq. 7 (Table III), two controls, and a four-outcome reading map (Table XIII) including a harness-calibration check at 128K tokens.
Honest negative reporting
Metric dissociations corroborating published ablations
Laptop-scale reproducibility
Weaknesses
Evidence-title scale mismatch
The Kimi-representative rung never learned the task
Seed variance overwhelms the central comparisons
Calibration circularity in the probe
Cross-family metric comparability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether gated attention and the hybrid linear-attention stack of Kimi K3 (69 of 93 layers fixed-size linear, 24 gated global layers, attention residuals, no positional encoding) eliminate attention sinks and uneven context reading at million-token windows. It delivers five things: the SinkProbe metric suite (Section III, Eqs. 1–4), a controlled four-model ladder of ~1M-parameter models, an objective-control experiment, an analytic cache model (Table VI), and a pre-registered protocol for released weights (Section VII, Tables III–IV, XIII).
The strongest claim is causal: the training objective, not the architecture, creates the sink. Table VII shows the identical eight-layer softmax stack with 31.6% first-token attention (worst layer 63.3%) under a per-position auxiliary loss versus 4.9% (worst layer 6.7%) when only the answer position is scored — with recall unchanged (32.0 vs 33.3%) and seed spread collapsing from {0.455, 0.089, 0.405} to {0.051, 0.046, 0.050}. Two further claims are dissociations: gating halves massive activations (μ 74.9 to 36.2) and attention residuals cut them 60% (97.9 to 38.9) while sink mass stays within seed noise; and cache arithmetic showing ~24 GiB vs ~2.5 TiB at 1M tokens. The title question itself is, by the authors' own conclusion (VI.K), left open pending the registered run.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did this paper actually test any model at one million tokens?
No. The experiments train about one-million-parameter models at 96 tokens and evaluate them only up to 768 tokens. The million-token part consists of an analytic cache-size calculation (about 24 GiB versus about 2.5 TiB for a dense stack of the same depth) and a pre-registered protocol intended to be run later on the released Kimi K3 weights.
What does the paper say causes attention sinks?
In the paper's small-scale control, the training objective causes them: the same softmax transformer put 31.6% of its attention on the first token when trained to predict at every position, but only 4.9% when trained to answer only the final question, with retrieval accuracy essentially unchanged either way.
Is the code and data for SinkProbe available?
Yes. The paper states that code, seeded synthetic data generation, and the measurement protocol are released on GitHub; everything runs on a laptop CPU without pretrained weights or a GPU, includes a test suite, and three scripts regenerate every table and figure in the paper.
What is the recency gap metric?
It is the difference between recall when the answer sits in the last quarter of the context and recall when it sits in the first quarter. A positive value means the model favors recent text; near zero means even reading. The paper reports the sign rather than assuming it, because the direction flipped between short and long evaluation lengths.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as captions in the extraction; plotted values cited in this review were taken from the tables and in-text descriptions, both of which were verified present in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.