The Automaton Underneath: The Additive Input Pathway Is a Parasitic Attractor for State Tracking in Householder Linear RNN
Gunner Levi Howe, Gunner Levi Howe
cs.LG · cs.NE
Quick verdict
Pre-registered ablation: deleting the additive input term flips shortcut collapse into exact state tracking in Householder RNNs; small-scale but disciplined.
Summary
The paper investigates why Householder-product linear RNNs (the DeltaNet/DeltaProduct class), though provably expressive enough for hard state-tracking, fail to length-generalize when trained. In a pre-registered, within-architecture causal ablation, deleting a single term — the additive input injection b_t = W_b e_t — flips the outcome: with it, models fit the 32-token training length and collapse out of distribution (median position-512 accuracy 0.20 on the non-solvable S5 word problem); without it, the same architecture reaches median accuracy 1.00 at 16× the training length. A tested representation law states that the minimal number of Householder factors per token equals the maximal reflection length of the task's generators in the representation pinned by the task format (parity 1, S4 3, A5/S5 4); below that width nothing fits, and the law's representation-relativity reconciles DeltaProduct's conflicting S4/A5 results at n_h=2. Two pre-registered arms locate the mechanism: from a verified-exact initialization with W_b=0, Adam grows the additive path and pulls the model off the exact solution, while zeroing W_b at inference restores exact generalization at law-minimal widths (18/20 seeds) — the additive pathway is 'parasitic', neither performing nor aiding the fit. Limitations: one architecture family and budget, d=112, five two-generator tasks, five seeds per cell, no inferential statistics, and mod-3 drifts even without b_t.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Deleting the single additive input term b_t = W_b e_t flips trained Householder linear RNNs from shortcut-and-collapse to exact length generalization (median 1.00 at 16× the training length) on parity, S4, A5, and S5.
- The minimal number of Householder factors per token needed for exact tracking equals the maximal generator reflection length in the task-format-pinned representation (parity 1, S4 3, A5/S5 4), and the law is representation-relative.
- From a verified-exact initialization with W_b=0, Adam actively grows the additive pathway (‖W_b e‖ up to 0.77 on S5) and destroys exact generalization, so the pathway is an attractor rather than a random-initialization basin effect.
- At law-minimal widths, zeroing W_b at inference on trained +b models restores exact generalization (18/20 seeds), showing the additive path conceals an already-learned automaton — it is 'parasitic'.
- Mod-3 drifts even without b_t (median 0.00 at n_h=4), refuting the authors' own transposition-anchor hypothesis; the locking-vs-drifting distinction remains open.
Strengths
Clean within-architecture causal design
Deleting only b_t = W_b e_t in a fixed DeltaProduct-style model isolates optimization from expressivity: Table 1 (§3) shows a full flip (S5 median pos-512 0.20 → 1.00) at matched width, budget, and data — the causal test the paper correctly says prior cross-architecture evidence (Terzić et al., Ebrahimi & Memisevic) lacks.
Pre-registration with self-falsification
Grid, predictions and kill criteria committed before any grid run (commits 2cf13e5 / 84e21c8); the registered prediction P3 was wrong, K2 fired as registered, and the 'leaky-counter route' story was withdrawn and replaced (§5) — falsification is reported prominently, not buried.
Tested representation law with a necessity side
Verified exact-init instrument
Boundary reporting and reproducibility
Weaknesses
No inferential statistics
Necessity side conflates representation with optimization
The 'parasitic' mechanism is partly interpretive
mod-3 anomaly is statistically odd and undiagnosed
Self-hosted pre-registration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This short note targets a sharply defined question raised by Shakerinava et al. [9]: why do Householder-product linear RNNs that provably represent hard state-tracking automata fail to length-generalize when trained? The author's answer is causal and single-term. In a fixed DeltaProduct-style architecture (d=112, n_h input-dependent Householder reflections plus a per-step linear readout), deleting only the additive input injection b_t = W_b e_t — so that input acts solely by selecting orthogonal transitions — converts a shortcut learner into an exact automaton learner. Headline numbers (Table 1, §3): with b_t, median position-512 accuracy is 0.20 on the non-solvable S5 word problem despite 1.00 at the training length 32; without b_t, median accuracy is 1.00 at position 512 (16× the training length) on parity, S4, A5, and S5. The paper couples this flip to (i) a representation law (§4): minimal n_h equals the maximal generator reflection length in the format-pinned representation (parity 1, S4 3, A5/S5 4), and (ii) a mechanism account (§5): the additive path is a 'parasitic attractor' that Adam grows even from a verified-exact initialization and that conceals an already-learned automaton recoverable by zeroing W_b at inference. Claims are explicitly scoped: no universality (the PD-SSM [14] counterexample is acknowledged), no language-modeling prescription.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
In Householder-product linear RNNs (the DeltaNet/DeltaProduct class), removing the single additive input term b_t = W_b e_t flips trained models from fitting the 32-token training length but collapsing on longer sequences to exactly learning the underlying automaton: median accuracy 1.00 at 16× the training length on parity, S4, A5, and S5 state-tracking tasks, versus 0.20 with the term present on S5.
Why do these models fail to length-generalize, according to the paper?
The additive input pathway acts as a 'parasitic attractor': it is the only channel through which training can inflate state norms, it grows even when training starts from a verified exact solution, it destabilizes that solution, and once the exact automaton is learned it conceals it. Zeroing the additive weights at inference restores exact generalization at minimal widths, so the automaton was present in the weights all along.
What is the 'representation law' in the paper?
The minimal number of input-dependent Householder factors per token needed for exact length generalization equals the maximal reflection length of the task's generators in the representation pinned by the task format — 1 for parity, 3 for S4, 4 for A5 and S5. Below that width models cannot fit at all. The law is representation-relative: DeltaProduct's S4/A5 results at n_h=2 used a compact SO(3) representation admitted by a different task format (group-element classification), so both results are correct.
Is the paper's result reproducible?
On paper, yes: the study is fully pre-registered with commit-stamped predictions and kill criteria, every number regenerates as a byte-verified macro from 202 per-run JSON artifacts, hyperparameters are fully specified, and the whole grid takes about 53 minutes on a single RTX 3080. Caveats: the pre-registration lives in the authors' own repository rather than a third-party registry, and the extracted text gives no public URL.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are available only as extracted text summaries; per-position curve shapes and seed bands could not be visually inspected.
- Per-cell min–max ranges and Appendix A details live in external run artifacts that were not available; Table 1 ranges are only summarized in prose.
- The arXiv ID and several 2026-dated citations could not be verified against external sources (offline review).
- Public accessibility of the pre-registration repository could not be confirmed from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.