LSTM-UT and Recurrent-Depth Transformers on Cellular Automata
cs.LG
Quick verdict
Compares BUT, CoTFormer, new LSTM-UT on Rule 30 extrapolation and delayed recall; LSTM-UT wins. Careful, well-controlled, but synthetic-only with few seeds.
Summary
This paper compares how recurrent-depth Transformers retain information across repeated applications of a shared block, using Rule 30 cellular automata and a delayed-recall benchmark. A Block Universal Transformer (BUT) carries only its latest hidden state; CoTFormer additionally caches attention keys/values from earlier repeats; the authors introduce LSTM-UT, which adds a bounded, LSTM-style gated cell state alongside the residual stream. On Rule 30, BUT extrapolates to unseen recurrent depths more reliably than CoTFormer (92.96% vs 50.04% cell accuracy at depth 9 after R4 training), though BUT itself eventually collapses with attractor-like latent convergence. A 2x2 factorial of clean-state injection and cache clearing shows CoTFormer's failure involves a state-cache interaction: correcting the state helps only with short retained history, while clearing the cache helps only when paired with a corrected state. In delayed recall, BUT outperforms CoTFormer, and attention/cache interventions show CoTFormer never learns query-dependent addressing of its cache. LSTM-UT achieves the best results on both tasks (99.97% cell accuracy at twice trained depth under R6 supervision; 90.39% vs BUT's 79.45% recall accuracy at horizon 12 with two recall applications) and leads 16/18 near-parameter-matched comparisons. Limitations: synthetic tasks only, 2-4 seeds sharing one data seed, historical baselines differ in checkpoint coverage, no code release, and the LSTM cell's internal mechanism remains unexamined.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A Block Universal Transformer extrapolates to unseen recurrent depths more reliably than CoTFormer on Rule 30, though its accuracy eventually collapses with attractor-like latent convergence.
- CoTFormer's depth-dependent failure arises from a state-cache interaction: correcting the current state helps only when retained history is short, and clearing the cache helps only when paired with a corrected state.
- CoTFormer does not learn query-dependent addressing of its cache; the nominally requested block is neither necessary nor sufficient for its delayed-recall answers.
- LSTM-UT, which adds a bounded gated cell state to a recurrent-depth Transformer, outperforms both baselines on Rule 30 extrapolation and delayed recall, retaining its advantage in 16 of 18 near-parameter-matched comparisons.
Strengths
Genuinely informative causal interventions
The 2x2 factorial of clean-state injection and cache clearing (Section 3.2.3, Table 5) dissociates the two failure components: the state-only effect falls monotonically from +0.182 (recent-2 cache) to -0.009 (recent-8/full), while the conditional cache-reset effect rises from +0.316 to +0.501 as the trained window grows — a clean interaction result, not just correlational accuracy curves.
Converging mechanistic evidence on CoTFormer's memory failure
Attention diagnostics (0.956% mass on the requested block at query 10), inference-time masks, the trained cache-policy sweep (Table 4), value permutation/hiding interventions (Table 6: hiding the requested block costs -0.01 pp; keeping only it costs -15.60 pp at query 19), and a single-head ablation (Table 7) all independently support the same conclusion that the cache is not used as addressable memory.
Parameter-matched controls for the new architecture
Unusually disciplined scoping and self-qualification
Diagnostics that separate failure modes
Weaknesses
Synthetic-only evidence base
Thin statistical power
Misaligned historical baselines behind the LSTM-UT headline
No code or data release
The proposed architecture receives none of its own mechanistic scrutiny
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks two questions about recurrent-depth Transformers — architectures that re-apply one shared block: (1) does the learned transition remain valid at more iterations than were supervised (depth extrapolation), and (2) can earlier intermediate states be retrieved after further computation (delayed recall)? It compares a Block Universal Transformer (BUT, current hidden state only), CoTFormer (adds an expanding key/value cache across repeats), and introduces LSTM-UT, which adds a bounded gated cell state. Claims are explicitly scoped to Rule 30 cellular automata and a delayed-recall extension; the authors disclaim transfer to language reasoning or world models. The four central claims — BUT extrapolates better than CoTFormer; CoTFormer's failure is a state–cache interaction; CoTFormer never learns query-addressable memory; LSTM-UT beats both — are each backed by dedicated experiments, and the hedging is unusually disciplined.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LSTM-UT?
LSTM-UT (LSTM Universal Transformer) is a recurrent-depth Transformer proposed in this paper. It applies one shared Transformer block repeatedly while an LSTM-style gated cell state — with input, forget, and output gates — persists alongside the residual stream. The memory stays fixed in size no matter how many recurrent steps run, unlike CoTFormer's growing key/value cache. On Rule 30 cellular automata and a delayed-recall benchmark, it outperformed both a Block Universal Transformer and CoTFormer.
What is the paper's main finding about CoTFormer?
Although CoTFormer caches attention keys and values from every recurrent step, it extrapolated to unseen depths less reliably than the simpler BUT, and in delayed recall it never learned to select the cached block matching the query: attention mass did not track the requested repeat, hiding the requested block changed accuracy by less than 0.03 percentage points, and keeping only that block cost up to 15.6 points. Its failure reflects an interaction between current state and retained history, not either component alone.
Is code available for this paper?
The paper text does not mention a code or data release and cites no repository. The synthetic tasks, architectures, and hyperparameters are described in enough detail to support reimplementation, but seed sensitivity means exact numeric replication is unlikely; reproducing the qualitative ordering (LSTM-UT >= BUT > CoTFormer) is the realistic target.
Do these results transfer to real LLM reasoning?
Not established. The authors explicitly scope all claims to deterministic binary cellular automata and their delayed-recall extension, noting these tasks remove perception, stochasticity, actions, and language. They position bounded gated memory as a candidate inductive bias for language reasoning and learned world models that requires direct future evaluation before any transfer claim.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 could not be analyzed (only captions and in-text numbers were extracted); visual claims about similarity heatmaps and attention distributions rest on the surrounding text.
- Cross-architecture comparisons reuse historical baseline runs whose checkpoint-selection coverage differs from the LSTM-UT protocol, as the authors disclose; exact re-analysis from the text alone is not possible.
- The reference list contains several 2026 arXiv identifiers (e.g., 2604.07822, 2605.26733, 2608.15062); these could not be verified without network access.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.