Stream Recursion Model (SRM)
Asael Sorensen, Charles Brock, David Chamberlain, Jennifer Minnich, Matthew Hoffman, Ramyaa Ramyaa
cs.LG
Quick verdict
SRM adds learned inter-stream attention to HRM, matching GPT-2 loss with fewer params but ~7x compute; promising, though single-run 5B-token evidence is thin.
Summary
This paper introduces the Stream Recursion Model (SRM), a modification of the Hierarchical Reasoning Model (HRM) aimed at mechanistic interpretability. The authors' core diagnosis is that HRM's additive inter-stream communication underutilizes each stream and prevents scaling beyond two streams. SRM replaces this with a learned 'connection function' — stream-wise attention in which each stream owns distinct query/key/value/output projections — so that routing between streams is learnable and directly analyzable. Trained on OpenWebText (~5B tokens, 10k steps), SRM-base (68M parameters) reaches a loss of 3.18 versus 3.20 for a same-configuration GPT-2 (124M); SRM-med (134M) reaches 3.08 and SRM-large (454M) 2.96, while a from-scratch HRM baseline stalls at 4.93. However, SRM needs roughly 6-7x the training compute of GPT-2, and the projection that SRM-large would reach 2.6-2.7 loss at 300B tokens is extrapolation from short runs. The analysis sections report hub-like stream routing under a 0.10 attention threshold, near-orthogonal connection outputs versus token inputs (cosine ~0.06, replicated at all scales), skewed stream-ablation importance (top 9 of 32 streams account for ~50% of causal impact), and coordinated mid-layer logit-lens degradation. Main limitations: single-seed runs, loss-only evaluation with no downstream tasks, a possibly mistuned HRM baseline, arbitrary routing thresholds, and no code release. The results are suggestive but preliminary.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Replacing HRM's additive inter-stream communication with a learned stream-wise attention 'connection function' enables HRM-style models to scale to more than two streams (§2, §3).
- SRM matches or beats a same-configuration GPT-2 with roughly half the parameters (SRM-base 68M, loss 3.18 vs 124M, 3.20) but needs ~7x the training compute on ~5B OpenWebText tokens (Table 1).
- A from-scratch HRM baseline fails on causal language modeling (loss 4.93), which the authors attribute to its additive stream-communication design (§5, §8).
- Stream-level analysis reveals consistent structure across three scales: hub-like routing streams, near-orthogonal connection outputs (cosine ≈ 0.056–0.067), and skewed ablation importance (top 9 of 32 streams ≈ 50% of causal impact) (§6, Appendices C–E).
Strengths
Clear architectural diagnosis with a minimal, analyzable fix
Section 2 pinpoints HRM's additive stream-communication as the bottleneck; the connection function (§2.1, §3.2.1) — per-stream Q/K/V/O projections forming a soft routing gate — is a simple remedy, and Figure 2's near-runnable PyTorch pseudocode plus Appendix B configs make the design unusually fully specified.
Honest compute and scale accounting
Table 1 openly reports the compute disadvantage (SRM-base 77h vs GPT-2 11h; ~7x) and §5 flags that only ~5B tokens were used; the authors also concede their 1,000-step hyperparameter sweeps are 'not representative' (Fig 3 caption).
Multi-scale replication of key analysis findings
Directly attacks a live disagreement in the literature
Negative results reported transparently
Weaknesses
Loss-only evaluation on short training runs
Single-seed runs with no variance reporting
HRM baseline is plausibly mistuned
Unsupported 300B-token extrapolation
Routing conclusions depend on an arbitrary threshold
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper introduces the Stream Recursion Model (SRM), a modification of the Hierarchical Reasoning Model (HRM, Wang et al. 2025) targeted at mechanistic interpretability. The diagnosis (§2): HRM communicates between its two streams by adding states at fixed intervals, which underutilizes streams and, the authors report, breaks on causal language data; hand-designing communication schedules for more streams is infeasible because configurations grow exponentially. The remedy (§2.1, §3): treat streams as network nodes whose edges are learned — implemented as stream-wise multi-head attention with per-stream Q/K/V/O projections (the 'connection function'), where keys/queries form a soft routing gate. Stated contributions (§1.2): (i) an HRM variant that scales with stream count; (ii) parity with vanilla transformers and large gains over HRM on language; (iii) stream-level analyses revealing routing hubs and stream specialization. The abstract's headline — 'comparable to GPT-2 on a per-parameter basis' — is supported by Table 1 but carries a compute asterisk the abstract omits: 7x (SRM-base) and 6x (SRM-med) training time.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Stream Recursion Model (SRM)?
SRM is a 2026 research architecture that modifies the Hierarchical Reasoning Model (HRM) to make it more amenable to mechanistic interpretability. It replaces HRM's simple additive communication between its two latent streams with a learned 'connection function' — attention across any number of streams, where each stream has its own query/key/value/output weights. This makes inter-stream routing learnable and directly analyzable, and lets the model scale beyond two streams.
Does SRM outperform GPT-2?
Partially. Trained on ~5B tokens of OpenWebText, SRM-base (68M parameters) reached a slightly better training loss than a same-configuration GPT-2 (124M): 3.18 vs 3.20, and larger variants reached 3.08 and 2.96. However, SRM needed roughly 6-7 times more training compute, and the results come from single short runs evaluated by training loss only, with no downstream benchmarks.
Why does the paper claim HRM fails on language modeling?
The authors trained HRM from scratch on OpenWebText and it only reached a loss of 4.93, far worse than GPT-2's 3.20. They attribute this to HRM's additive stream-communication design. Reviewers note the HRM baseline used a 10x smaller learning rate and puzzle-task hyperparameters, so the failure may partly reflect tuning rather than architecture alone.
Is SRM code available?
The paper does not mention a code release. It does provide detailed PyTorch-style pseudocode (Figure 2) and full configurations in Appendix B, so reimplementation is feasible, but reproducing the training runs requires on the order of 70-260 GPU-hours per model on L40S-class hardware.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (routing heatmaps, training curves, alignment plots) are available only as captions and embedded textual descriptions; the visual claims built on them could not be independently inspected.
- Table 1 reports 'Training Loss' without specifying validation protocol or checkpoint selection; per-parameter comparisons assume the numbers are directly commensurable across models.
- Whether the token-wise attention inside step functions is causal is not shown in the pseudocode; the review assumes standard causal masking for a language-modeling setup.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.