Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing
Hao Li, Yasuyuki Tahara, Yuichi Sei
cs.LG
Quick verdict
Preregistered probing study showing older MoE expert selections predict routing beyond the previous layer; controls are careful, but small scale and a single corpus temper generalization.
Summary
This empirical probing study asks whether a sparse mixture-of-experts (MoE) router's immediately preceding expert selection fully summarizes a token's selection trajectory for predicting the next router. The authors freeze two pretrained models (OLMoE-1B-7B, JetMoE-8B), route WikiText-103 blocks, and fit ridge probes and small MLPs to predict target-layer router logits from k preceding binary selection vectors, always keeping the previous layer as baseline. Extending history from 1 to 11 layers raises OLMoE held-out R2 from 0.59879 to 0.66544; a preregistered JetMoE replication at target layers 12 and 20 gives four-layer gains of 0.14275 and 0.20528, with paired bootstrap CIs excluding zero. Controls address the main confound: MLP decoders of the recent state alone add only 0.00139/0.00936 over ridge, whereas adding history adds roughly 0.17-0.22; parameter-matched recent-only MLPs and cross-fitted residual prediction (R2 0.20549/0.23556) preserve the effect. Earlier experiments show selection identity, not contribution content, carries the accessible signal. A closed-loop policy experiment (XEC) honestly failed to convert this structure into next-token NLL gains. Limitations: small held-out sets (256-1024 samples), one token position per block, a single corpus, PCA budgets that change with window size, and unassessed pretraining overlap between split blocks.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Expert-selection identity — not fused expert outputs or contribution content — is the dominant accessible signal for predicting later routing (EIPC/EPD, Table 1).
- Older expert selections add held-out predictive value beyond the immediately preceding selection in OLMoE (R2 0.59879→0.66544 as k goes 1→11) and in preregistered JetMoE replication (Δ4 = 0.14275 and 0.20528 with bootstrap CIs above zero).
- The history gain is not an artifact of linear probing: nonlinear, parameter-matched, and cross-fitted residual controls all preserve it (NHD, Table 3).
- This predictive structure did not translate into routing-control benefit in the XEC five-action policy experiment, which showed no reliable NLL improvement.
Strengths
Explicit previous-layer baseline on both sides of every comparison
The core metric Δl(k)=Rl(k)−Rl(1) (Sec. 2.1) always conditions on the immediately preceding selection, so the claim is genuinely incremental value of older layers rather than generic cross-layer predictability — a cleaner question than most routing-dynamics probes.
Preregistered replication on a second architecture
JetMoE targets (L12, L20) and the frozen k=4 decision rule (positive baseline, Δ4≥0.02, CI lower bound above zero) were set in advance; 10,000-resample paired bootstraps exclude zero at both targets (Table 2), and the router identity was verified by exact recomputation (max abs logit difference 0).
Controls directly target the linear-decoding confound
Honest negative result on actionability
Signal-source decomposition before the headline claim
Weaknesses
Diminishing and modest absolute gains at deeper histories
Small, reused evaluation captures
Narrow input diversity
PCA budget confound across history windows
Pretraining overlap between FIT and TEST unassessed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
In frozen sparse mixture-of-experts models (OLMoE and JetMoE), expert choices from earlier layers still improve prediction of a layer's router beyond what the immediately previous layer's choice provides — by up to about 0.07 R2 in OLMoE and 0.14-0.21 in JetMoE — and the gain survives nonlinear decoders, parameter-matched controls, and residual analysis.
Does this mean MoE models can be made more efficient using routing history?
Not yet. The paper's control experiment (XEC) tried a five-action routing policy on OLMoE and found no reliable next-token loss improvement; the authors show predictive structure exists in routing trajectories but were unable to convert it into control gains, and they explicitly frame actionability as unresolved.
Which models and data were used?
Frozen OLMoE-1B-7B and JetMoE-8B evaluated in FP16, routing blocks from the WikiText-103 training split; probes are trained on held-out blocks with preregistered target layers 12 and 20 for JetMoE, and each 129-token block contributes one token at position 127.
Is the code available for this study?
Yes. The paper links a code archive (github.com/withfanta/moe-routing-dynamics) that includes full model identifiers, revisions, seeds, and experimental protocols, giving it high reproducibility.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (if any) are not recoverable from the text extraction; assessment relies on Tables 1-3 and in-text numbers.
- The XEC experiment is described only within the Discussion, with no methods section, so its setup cannot be fully audited from this text.
- Protocol details live in the linked code archive ('in the linked code archive'), which was not inspected; sample-size and split claims are taken from the paper text.
- Abstract R2 values match Table 2 exactly, suggesting the extraction is near-complete, but possible omitted appendix content cannot be ruled out.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.