Memory Attention
cs.LG
Quick verdict
Replaces attention's value projection with token-ID memory plus keys; gains look real but are confounded by 2-3x added params; no param-matched baseline.
Summary
Memory Attention (MA) replaces the value projection in self-attention with V = K + Norm(M), where M is a per-layer embedding table indexed by token ID and K is the pre-RoPE contextual key; at inference the head-wise normalization folds into the table, reducing value construction to lookup plus addition. Because lookup addresses depend only on token IDs, MA-Offload stores folded tables in CPU memory and prefetches entries; MA-Recall sketches value reconstruction from retained keys without a value cache (analyzed, not measured). The author trains 24-layer models (hidden 1,024: MHA/GQA/MQA; hidden 2,048: MHA) on FineWeb-10BT for 10-20B tokens. At matched token budgets MA improves perplexity (e.g., GQA LAMBADA 50.46 to 43.68) and average zero-shot accuracy (deltas +0.23 to +1.16 points), wins all nine single-needle retrieval tests including 2x context extrapolation (mean 41.9% vs 25.9% at 4K), and MA-Offload keeps latency within +/-3% of baseline while cutting GPU parameter storage 7.38% below Standard despite 2.08x total parameters. Training FLOPs, however, rise 1.78x, so at the larger scale total compute per matched loss is roughly 1.5x higher. The paper candidly admits the gains are not isolated from added capacity (parameters grow 1.57-3.04x); there is no parameter-matched baseline, no comparison with the cited Value Embeddings/PLE/STEM/Engram, and results are single-seed without variance reporting.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Replacing the dedicated value projection with V = K + Norm(M) (layer-specific token-ID memory plus contextual keys) improves perplexity and average zero-shot accuracy at matched training-token budgets across MHA, GQA, and MQA configurations and a larger hidden-2,048 model.
- At inference, folding head-wise RMSNorm into the memory tables reduces online value construction to lookup plus addition.
- Token-indexed retrieval enables CPU offloading with prefetching: MA-Offload holds 2.08x total parameters with 7.38% less GPU parameter storage than Standard while keeping forward latency within +/-3%.
- MA reaches matched loss with 1.42x (hidden 1,024) and 1.16x (hidden 2,048) fewer training tokens.
- Token memory improves single-needle retrieval on all nine task-length combinations, including 2x context-length extrapolation (mean 41.9% vs 25.9% at 4K).
Strengths
Crisp, fully specified mechanism
V = K + Norm(M) with per-layer tables and head-wise RMSNorm is pinned down in §3.1 (Eqs. 7-9) and Appendix A (Table 6 memory configuration), including the RoPE interaction (values built from pre-RoPE keys) and the folded-norm inference path (Eqs. 16-17).
Consistent directional evidence across configurations
All four Standard-MA pairs improve both PPLs and average accuracy (Table 2: +0.23 to +1.16 points), and MA wins all nine NIAH cells with large margins (in-window means 97.4% vs 82.6% at 1K; Table 3).
Honest scope discipline
Working systems prototype with transparent accounting
Useful cost analysis
Weaknesses
Central confound acknowledged but never tested
Missing baselines and ablations
Net training compute is unfavorable and under-discussed
No inferential statistics
Scalability of vocab-sized tables unexamined
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Memory Attention (MA) is a single-author method paper proposing to delete the value projection W_V from self-attention and build values as V = K + Norm(M), where M is a per-layer embedding table indexed by token ID (E in R^(N x dv), N = 32,000 here) and K is the pre-RoPE content key (§3.1, Eqs. 7-9). Three derived claims: (1) quality holds or improves at matched training-token budgets; (2) at inference the head-wise RMSNorm folds into the tables (Eqs. 16-17), reducing online value construction to lookup plus addition; (3) token-ID-only addressing enables CPU offloading with prefetch (MA-Offload, §3.4), plus an analyzed-but-unmeasured value-reconstruction mode (MA-Recall, §3.5). A functional comparison to MLA (§3.2) motivates direct key reuse and is carefully disclaimed as non-equivalent. The paper is unusually candid about its central confound: the results "do not isolate the contribution of its structure from the increase in parameter capacity" (§4.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Memory Attention?
Memory Attention (MA) is a transformer attention variant that removes the value projection matrix and constructs attention values by adding a per-layer, token-ID-indexed memory table to the attention keys: V = K + Norm(M). The keys keep context dependence; the memory supplies token-specific content learned independently at each layer.
Does Memory Attention reduce GPU memory usage?
Its offload variant does, modestly: the paper reports GPU parameter storage 7.38% below a same-backbone standard model (2,410 vs 2,602 MiB) even though total parameters are 2.08x larger, because the vocab-sized memory tables (3,000 MiB) live in CPU RAM and are prefetched. Total memory across devices grows, and 3 GB of host RAM is required.
Is Memory Attention faster or slower than standard attention?
At the tested single-H800 workload (batch 8, 2K context), prefill was 3.04% faster and decode 0.86% slower with tables on GPU; with CPU offloading, decode was 2.53% faster. The author cautions these small deltas are specific to this operating point and may not generalize.
What is the paper's main limitation?
MA multiplies parameters by 1.57-3.04x via vocab-sized per-layer tables, and no parameter-matched baseline is trained, so the quality gains cannot be attributed to the architecture rather than raw extra capacity - a limitation the author explicitly acknowledges.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are recoverable only as embedded text; the loss curves underlying the 1.42x/1.16x token-efficiency ratios and the Figure 3 error bars cannot be visually inspected.
- Extraction appears complete through Appendix A, but figure panels (e.g., the architecture diagram, Figure 2 sub-panels) are represented only partially in text.
- Two key related works (Engram [3], STEM [11]) are 2026 preprints that could not be verified offline; the novelty assessment relies on the paper's own characterization of them.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.