Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
cs.LG · cs.CL
Quick verdict
Decomposes recurrent recall into convolution/transition/training effects; curriculum breaks interference wall. Rigorous, toy-scale, mostly well-supported.
Summary
This paper decomposes associative-recall performance of fixed-state recurrences (linear attention and state-space cells) at a matched state budget of 1,024 elements, toggling three knobs one at a time: a short causal convolution, the transition structure (rank-1 delta rule vs. diagonal), and decay. Under matched training at K=32 the convolution dominates, adding roughly +0.45-0.5 recall in both families; with per-arm learning-rate tuning the delta-family gain compresses to ~+0.13. The rank-1 transition beats its own diagonal ablation by +0.19/+0.32 at K=16/32 (10 seeds), but the margin shrinks to +0.034 once both cells carry the convolution, and a state-matched Mamba-2 ties or beats the unarmed rank-1 cell, so no architecture-class claim survives. Decay shows no measurable cost (p=0.86, n=20). Cells that solve 32-pair recall drop to chance retrieving 4 pairs across distractors, flat over lengths; the authors attribute this to interference under sparse supervision, not capacity: a distance curriculum lifts the unchanged architecture from 0.021 to 1.000 (lock-in 1/10 to 7/10, p=0.02), and shaped or success-gated ramps extend the fix to L=256 (4/5 vs. 0/9) and L=512 (6/6 vs. 0/6). Bidirectional denoisers show no directional advantage; arming costs nothing on S5 state tracking. Limitations: toy scale (d in {32,128}), one synthetic task family, a PyTorch Mamba-2 reference that under-trains the official kernel, and scale evidence deferred to a companion paper.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- At matched state (1,024 elements), the short causal convolution is the dominant recall lever (+0.47 delta-rule, +0.44 diagonal at K=32 under matched training), so comparisons pitting convolution-free cells against convolution-equipped Mamba measure the wrong ingredient.
- The rank-1 transition's genuine within-cell margin (+0.19/+0.32 at K=16/32 without convolution) shrinks to +0.034 once both cells carry the convolution, and a state-matched Mamba-2 ties or beats the unarmed rank-1 cell — no architecture-class claim survives.
- The haystack failure (chance on 4 pairs, flat across lengths and transition types) is interference under sparse supervision, not capacity or decay: a distance curriculum takes the unchanged architecture from 0.021 to 1.000.
- Training in this regime is a lock-in lottery whose rate curriculum shape controls: 1/10 to 7/10 at L=64 (p=0.02), 4/5 vs 0/9 at L=256, and 6/6 vs 0/6 at L=512 with a success-gated ramp (p=0.0011).
- Bidirectional denoisers read the query first natively (JRT property) but show no measurable recall advantage over causal training at 10 seeds; any circuit exploiting the property needs two layers.
Strengths
Clean single-knob factorial at matched state
True within-cell toggles (dg rank vs dg diag differ only by setting the rank-1 term to zero; mamba2 ref noconv sets conv width to 1) at a fixed budget of 1,024 state elements isolate each ingredient, directly resolving the confound in whole-architecture comparisons (§3-§4, Tables 1-3).
Statistical hygiene above field norm
Rebinding controls verify genuine key-value binding (Table 3 rebound 0.042-0.073 vs trained 0.62-0.85), floors quoted with every result (Appendix C), Fisher exact on lock-in rates for bimodal seeds, and a learning-rate sensitivity sweep that directly answers the Okpekpe & Orvieto learnability confound (§3.4, §4.2).
Honest negative results and self-correction
Exceptional reproducibility scaffolding
Weaknesses
Toy scale, single task family
Internal inconsistency on decay
Comparator under-trains the official kernel; unexplained cross-table discrepancy
Heterogeneous seed budgets and pooling choices
Scale claims outsourced to an unavailable companion
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks which ingredient of modern fixed-state recurrent cells causes their associative-recall deficit relative to attention. It decomposes masked MQAR performance at a fixed state budget (1,024 state elements, ~29k parameters, d=32, one layer) along three single-knob axes: a width-4 short causal convolution, transition structure (rank-1 delta rule vs. diagonal), and decay, across seven arms (Table 1). Central claims: (1) the convolution is the dominant lever (+0.47 delta-rule, +0.44 diagonal at K=32 matched-training; ~+0.13 tuned in the delta family); (2) the rank-1 transition's within-cell advantage (+0.19/+0.32 at K=16/32, 10 seeds) shrinks to +0.034 (p=0.0023) once both cells are armed, and a state-matched Mamba-2 ties or beats the unarmed rank-1 cell, so no architecture-class claim survives; (3) decay has no measurable recall cost (p=0.86, n=20); (4) the haystack failure (chance on 4 pairs, flat over L=64-256, transition-independent) is interference under sparse supervision, not capacity; (5) a distance curriculum lifts an unchanged cell from 0.021 to 1.000, shifting lock-in from 1/10 to 7/10 (p=0.02).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do Mamba and other SSMs fail associative recall compared to attention?
This paper argues the deficit is mostly not the recurrence itself: under matched training, adding a short causal convolution to a recurrent cell raises recall by roughly +0.5 at matched state, and a training-only distance curriculum fixes a distractor-interference failure on an unchanged architecture (0.021 to 1.000 on one task). Whole-architecture comparisons largely measure the missing convolution and the training signal, not the state transition.
What is the haystack or interference wall in linear-attention and state-space models?
It is the finding that recurrent cells able to store 32 key-value pairs at near-perfect accuracy still drop to chance when retrieving just 4 pairs after a stretch of distractor tokens, at every tested sequence length. The paper shows this is a training-coverage problem (sparse supervision gives the write gate no gradient), since a curriculum over table-to-query distances solves it without architectural change.
Does the delta rule actually beat diagonal SSM transitions at recall?
Only within the same cell and without a convolution: +0.19/+0.32 at 16/32 key-value pairs (10 seeds). Once both cells carry the short convolution the margin shrinks to +0.034, and a state-matched Mamba-2 ties or beats the unarmed rank-1 cell, so the paper concludes no ranking of architecture classes is supported.
Is the paper's code and data available?
Yes. The paper states that code and result JSONs are public at github.com/JIBSIL/dualgoose, with exact per-table reproduction commands in Appendix B; everything runs on commodity GPUs, with the headline experiments executed on a single GTX 1070.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted text with axis labels and bar values; visual details such as error bars and per-seed trajectories could not be inspected.
- The companion preprint (Boesch & Wee 2026) carrying the 1.7B/8B scale claims was not available for review; its results are taken on trust.
- Review is based solely on the v1 preprint text; no external verification of the GitHub repository contents was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.