It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Raito Kiya, Satoki Ohashi, Kosuke Sato, Go Kamoda, Ryosuke Takahashi, Yuji Yamamoto, Daiki Shiono, Keisuke Sakaguchi, et al.
cs.CL
Quick verdict
Argues attention sinks and massive activations arise from causal-mask self-concentration and Value-non-mixing, not RoPE; 5-LLM interventions support it.
Summary
The paper asks why attention sinks (AS) and massive activations (MAs) form at the first sequence position in LLMs regardless of which token occupies it. The authors decompose initial-position properties into three factors — BOS token identity, the first RoPE positional encoding, and forced self-concentration under the causal mask — and test each by intervention on five open-weight models (Llama-2-7b-hf, Llama-3.2-3B, Mistral-7B-v0.3, Qwen2-7B, pythia-1b) over WikiText. Swapping BOS with a mid-sequence token (Section 3, Table 1) shows token identity and position each contribute. Re-indexing only the first token's key-side RoPE (Section 4.1, Table 2) barely changes Sink_ε, so RoPE is not the primary driver. Forcing α16,16 = 1.0 at all layers (Section 4.2, Table 3) induces AS and elevated hidden-state norms at a non-initial position in all five models, weakest for Mistral. The paper proposes Value-non-mixing — outputs passing through a single Value vector — as the underlying state, tested with repeated-token sequences containing one inserted distinct token (Section 4.3, Table 5): AS and MAs appear exactly where all attended Values are identical, clearly in three of five models, weakly and input-dependently in Mistral and pythia. Ablations span datasets, lengths 32–256, positions 8/16/32, layer subsets, and thresholds. Limitations: five RoPE-only models, no quantization-error evaluation, and largely visual MA evidence.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Attention sinks and massive activations at the initial position arise from causal-mask-enforced self-concentration (α1,1 = 1.0), not from RoPE or BOS identity alone.
- Forcing self-concentration at a non-initial position (t=16, all layers) induces both AS and MAs in all five models tested, weakest in Mistral-7B-v0.3.
- Value-non-mixing — the attention output passing through a single Value vector — is the proximal state underlying AS and MAs, supported by repeated-token experiments in three of five models.
- Re-indexing only the first token's key-side RoPE does not attenuate the sink (Table 2), so key-side positional encoding is not the primary driver.
Strengths
Clean factorial decomposition of the initial position
Section 3 separates BOS identity from position (Table 1: with BOS moved to position 16, Llama-3.2-3B Sink rises from 0.0000 to 0.4312 at position 16 while non-BOS position 1 still reaches 0.8856), and Sections 4.1-4.2 then isolate RoPE versus causal-mask self-concentration with targeted interventions.
Converging interventions triangulate the mechanism
Forced self-concentration on natural text (Section 4.2) and repeated-token sequences that induce Value-non-mixing without clamping (Section 4.3) attack the same hypothesis from two directions; Table 5 shows Llama-3.2-3B Sink jumps from 0.0000 (uniform) to 0.7740-0.8042 (distinct token inserted) exactly where all attended Values are identical.
Unusually thorough ablation program
Honest reporting of heterogeneous results
Weaknesses
Title claim exceeds the tested intervention
Massive activations lack a quantitative metric
Out-of-distribution confound in the repeated-token design
No statistical inference anywhere
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a mechanistic question: why do Attention Sinks (AS) — heads concentrating attention weight on the first token — and the accompanying Massive Activations (MAs) — a few extremely large hidden-state values — form at sequence position 1 regardless of which token occupies it (Section 1)? The authors decompose the initial position into three candidate causes: BOS token identity, the first positional encoding (RoPE), and causal-mask-enforced self-concentration (α1,1 = 1.0). Their central claim is that self-concentration, acting through a state they name Value-non-mixing (the attention output reduces to a single Value vector, Equation 4), contributes to both AS and MAs, and that RoPE is not the driver. Claims are framed as empirical contribution (contribute to) rather than complete explanation, which is appropriate.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What causes attention sinks in LLMs, according to this paper?
The paper argues that the causal attention mask forces the first token to attend only to itself (attention weight 1.0), a state the authors call self-concentration. Under this condition the first token's attention output is a pure pass-through of its own Value vector (Value-non-mixing), which their interventions show contributes to attention sinks and the accompanying massive activations at the initial position, regardless of which token occupies it.
Is RoPE responsible for attention sinks?
No, according to this study. Re-indexing only the first token's key-side RoPE rotation left sink strength nearly unchanged in all five tested models — for example Llama-2-7b went from 0.9305 to 0.9273 (Table 2). The authors caution they tested only the key side, not the query side, so the conclusion is scoped to that intervention.
What is Value-non-mixing?
It is the condition where all Value vectors receiving nonzero attention from a query are identical, so the attention output is simply that single Value vector passed through the output projection, with no information mixed from other tokens. The first token under a causal mask is always in this state, and repeated-token experiments show sinks and massive activations form exactly where this state holds.
Does this paper improve quantization of LLMs?
Not directly. Massive activations complicate low-bit quantization, which motivates the study, but the paper runs no quantization-error, perplexity, or accuracy experiments; it explicitly states that establishing a quantization connection requires dedicated future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Quantitative values inside figures (heatmap patterns, norm-curve shapes) cannot be verified from the extracted text; assessment of the MA evidence relies on captions, author descriptions, and the tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.