RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
Luca Herranz-Celotti, Vincent Guigue
cs.LG · cs.AI
Quick verdict
Order-3 tensor state generalizes linear attention; MQAR results strong, but claimed post-pretraining downstream gains lack data in the provided text.
Summary
The paper proposes RunningTensor, a linear-time causal sequence model that generalizes linear attention's matrix-valued recurrent state to an order-o tensor, instantiated at order 3. The state accumulates rank-1 outer products v⊗k⊗r, is read by contracting against two query vectors, and retains both recurrent and chunk-parallel forms; Proposition 1 shows the order-2 special case recovers linear attention. The full model adds GLA-style data-dependent forget gates, query-skip gates that let readouts use both, one, or neither tensor axis, a third-order tensor delta rule, and ConcOnlyC, a near-parameter-free fused projection. Models are trained with 0.6B parameters on FineWeb-Edu 100BT. On multi-query associative recall (MQAR) at d_model=128, RunningTensor (176k parameters/layer) reaches 100% F1 at lengths 1024–4096, versus Gated DeltaNet 61.2 average, Mamba-2 0.2, and GLA 0.7, with faster forward passes than attention (311.9 vs 333.5 ms/step). Ablations attribute the gains mainly to the tensor state plus delta rule; matrix-state variants collapse (20.7–30.6 average). Key limitations: the claimed post-pretraining improvements on language-understanding and retrieval tasks have no supporting data anywhere in the provided text (§3 covers only MQAR; the Conclusion section is empty), the MQAR comparison confounds state capacity since the tensor state is O(W³) versus O(W²) for baselines, and no variance or significance statistics are reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Linear attention's matrix state generalizes to an order-o tensor updated by rank-1 outer products and read via o−1 query contractions, with order 2 recovering linear attention exactly (Proposition 1).
- The order-3 instantiation retains both recurrent and parallel forms while keeping cost linear in T and expanding state size from O(W²) to O(W³).
- On synthetic MQAR at d_model=128, RunningTensor (176k params/layer) reaches 100% F1 at 1024–4096 tokens, outperforming linear-attention and SSM baselines.
- After 0.6B-parameter pretraining on FineWeb-Edu 100BT, the model improves language-understanding and non-synthetic retrieval tasks — a claim with no supporting data in the provided text.
Strengths
Principled generalization with containment guarantee
Proposition 1 (§2.2) proves the order-2 case with dr=1 reduces exactly to causal linear attention, and both recurrent and chunk-parallel (UT/WY) forms are retained, so the model plugs into established linear-attention machinery rather than replacing it.
Large synthetic recall margins at favorable parameter cost
Table 1: 100% F1 at 1024/2048/4096 tokens with 176k non-embedding parameters per layer and 311.9 ms/step, versus Attention at 934k params and 333.5 ms/step, and Gated DeltaNet averaging 61.2 F1.
Granular component ablation
Unusually complete implementation detail
Weaknesses
Headline downstream claims unevidenced in the text
Capacity confound in the MQAR comparison
Latency claim contradicts own table
Thin statistical reporting
Single small controlled scale
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RunningTensor (arXiv 2609.12814, cs.LG) is a method paper on efficient sequence modeling. The proposal: lift linear attention's matrix-valued recurrent state (an order-2 tensor) to an order-o tensor, instantiated at o = 3. The state accumulates rank-1 outer products v⊗k⊗r and is read by contracting against two query vectors (q, u); recurrent and chunk-parallel forms coexist and cost stays linear in T (§2.2). Proposition 1 proves containment: with dr = 1 and u = r = 1 the readout collapses exactly to causal linear attention. The abstract makes two empirical claims: (i) RunningTensor beats linear-attention and SSM baselines on synthetic multi-query associative recall (MQAR); (ii) after pretraining it improves language-understanding and non-synthetic retrieval tasks. Claim (i) is supported by §3; claim (ii) is not supported anywhere in the provided text — §3 contains only MQAR (§3.1) and component ablations (§3.2), and §5 Conclusion is empty in the extraction.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RunningTensor?
RunningTensor is a linear-time causal sequence model that generalizes linear attention's matrix-valued recurrent memory to a third-order tensor: it accumulates rank-1 outer products of three vectors and reads the state by contracting it with two query vectors. The order-2 special case reduces exactly to linear attention (Proposition 1), and both a recurrent form and a parallel training form are retained.
How does RunningTensor perform on the MQAR benchmark?
At d_model=128 it reports 100% validation F1 at sequence lengths 1024, 2048 and 4096 with 176k non-embedding parameters per layer, versus an average of 61.2 for Gated DeltaNet, 0.2 for Mamba-2 and 0.7 for GLA. Its forward pass (311.9 ms/step) is faster than attention (333.5 ms/step) but slower than the reported SSM baselines.
Does RunningTensor beat Mamba or Gated DeltaNet?
On the paper's synthetic multi-query associative recall benchmark, yes, by large margins. The abstract also claims improvements on language-understanding and non-synthetic retrieval tasks after 0.6B-parameter pretraining on FineWeb-Edu 100BT, but the provided text contains no tables or numbers backing those downstream claims (Section 3 covers only MQAR and ablations; the Conclusion section is empty).
Is code available for RunningTensor?
No code or data release is mentioned in the paper. The architecture, gate parameterizations (e.g., tau=16), fused-projection design, and training setup are described in enough detail for reimplementation, and MQAR is a public benchmark from the Zoology project.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- No post-pretraining downstream results (tables or numbers) appear anywhere in the provided text: §3 holds only MQAR and ablations, and §5 Conclusion is empty in the extraction. If an appendix after the references was lost in extraction rather than absent from the paper, this criticism must be re-verified against the source PDF.
- Supplied metadata lists two authors, but the paper text lists three (Luca Herranz-Celotti, Ermal Rrapaj, Vincent Guigue; ISIR Sorbonne / Berkeley Lab); the review treats the in-text list as authoritative.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.