Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference
Hongjin Lin, Wentao Wan, Keze Wang
cs.AI · cs.CL
Quick verdict
HeRo adds linear-attention memory to dynamic FFN routing; strong 3-backbone results, but no latency data and >100% retention partly reflects SFT gains.
Summary
The paper argues that token-level dynamic layer routing is a path-dependent sequential decision process: earlier gate choices shape the hidden states and residual updates seen by downstream routers, yet existing routers condition only on the current hidden state. It proposes HeRo, which augments each router with a memory built by kernelized linear attention over model depth, incrementally aggregating preceding routing scores, residual-stream transitions, and progress features into a compact state. At each routed layer, HeRo chooses between the frozen pretrained FFN and a small bottleneck adapter, training only routers and adapters on a frozen Llama backbone with one epoch of full Tulu-3 SFT. Across Llama 3.1-8B, 2-7B, and 2-13B at roughly 25% and 40% parameter-skipping budgets, HeRo reports the highest aggregate retention among ten pruning/routing baselines: 100.24% of dense performance at 26.87% skipping on Llama 3.1-8B, and 97.01% at 38.82%. Ablations attribute the largest history contribution to GSM8K and HumanEval. Limitations: efficiency is measured only by parameter counts, with no latency, throughput, or FLOP numbers; the above-100% headline partly reflects fine-tuning gains (the LoRA control scores 99.87%); reasoning/code benchmarks are not evaluated for baselines; and no code release is stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 2
● low 4
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Dynamic routing across model depth is a path-dependent sequential decision process, and the current hidden state alone is an incomplete observed state for routing because it does not explicitly record preceding gate choices.
- A router memory built by kernelized linear attention over depth, aggregating prior gate scores and residual-stream updates, yields higher performance retention than purely local routers at matched parameter-skipping rates.
- On Llama-3.1-8B, HeRo bypasses 26.87% of parameters while achieving 100.24% of dense performance across seven benchmarks, and retains 97.01% while bypassing 38.82% under a tighter budget.
- Ablations attribute the largest contribution of the routing history to multistep reasoning (GSM8K) and code generation (HumanEval).
Strengths
Well-motivated problem formulation
The Introduction cleanly characterizes routing as a path-dependent sequential decision process: prior gates determine which residual updates reach downstream routers (Eq. 1-3), and the utilization loss (Eq. 18) couples decisions across depth — hence the current hidden state is an incomplete observed state. The argument is grounded in residual-stream (Shai et al. 2024) and recurrent-MoE-router (Qiu et al. 2025) evidence.
Broad, controlled comparison
Table 1 compares ten baselines across three backbones (Llama 3.1-8B, 2-7B, 2-13B) at 25% and 40% target budgets, reports realized skipping rates for every method, and includes a LoRA r=128 SFT control (99.87% retention on 8B) that contextualizes fine-tuning gains — an unusually honest reference row.
Clean mechanism ablation
Practical and fully specified training recipe
Realized-skip reporting favors conservatism
Weaknesses
No wall-clock efficiency evidence
Above-100% retention conflates SFT gains with routing quality
Reasoning/code advantage untested against baselines
Conspicuous baseline omissions
No code or model release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper targets token-level dynamic FFN routing in decoder-only LLMs. The core thesis (Introduction) is that routing along depth is a path-dependent sequential decision process: earlier gate choices determine which residual updates reach downstream routers (Eq. 1-3), and the utilization loss (Eq. 18) couples decisions jointly — yet existing routers condition only on the current hidden state, an "incomplete observed state." The remedy is HeRo: a kernelized linear-attention memory over depth (Eqs. 10-12, read-before-write) that aggregates prior gate scores, residual-transition magnitude/cosine features, and progress scalars (Eq. 4), whose history logit is summed with a local logit and thresholded at 0.5 (Eq. 16). Three claims: (1) explicit routing history improves routing quality; (2) best aggregate retention among ten baselines on three Llama backbones at ~25% and 40% parameter-skipping budgets; (3) history matters most for multistep reasoning and code.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does HeRo solve?
In dynamic layer routing for LLMs, each layer's router normally decides using only the current token representation. HeRo argues routing is a sequence of linked decisions across depth — earlier skip choices change what later routers see — and adds a lightweight memory (linear attention over depth) that lets each router condition on the accumulated routing history plus the current representation.
How much faster is HeRo than a dense LLM?
The paper reports parameter skipping, not measured speed: on Llama-3.1-8B it bypasses 26.87% of parameters at the standard setting and 38.82% under a tighter budget. No latency, throughput, or FLOP numbers are given, so real-world speedup is not established by the paper.
Does HeRo lose accuracy compared to the full model?
The paper reports 100.24% of dense-model performance averaged over seven benchmarks on Llama-3.1-8B at 26.87% skipping, and 97.01% at 38.82%. Note a caveat the paper's own control reveals: fine-tuning alone with LoRA reaches 99.87%, so most of the above-100% figure comes from fine-tuning rather than from routing quality.
Is HeRo's code available?
The paper does not mention any code or model release. Training details (Tulu-3 SFT data, hyperparameters, seeds, hardware) are described thoroughly, but replication would require a new implementation and multi-GPU training.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 appear only as captions in the extracted text; their visual content could not be inspected. All tables, equations, the abstract, and the reference list appear complete in the provided extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.