From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts
Kang Chen, Sihan Zhao, Yixin Cao, Yu-Gang Jiang
cs.AI
Quick verdict
Proposes routing effective rank: a reproducible low-high-low arc in MoE reasoning cohorts; rigorous stats and honest nulls, but thin gain over plain density.
Summary
The paper introduces routing effective rank (d_eff), the entropy-effective dimensionality of a cross-rollout similarity graph built from MoE router gate weights, as a label-free monitor of reasoning-cohort organization. Sampling 64 rollouts per question over ten MoE configurations (gpt-oss-20B/120B, Qwen3-30B-A3B, Qwen3-Next-80B at several reasoning-effort tiers) and five math/science benchmarks (3,105 cohorts), the authors find a reproducible low-high-low d_eff trajectory with a prominent interior maximum in 98.5% of cohorts. An exact spectral decomposition log d_eff = h(m) + (1-m)log d_perp attributes roughly two thirds of the trajectory to common-mode mass and one quarter to residual dimensionality. Common-mode concentration strongly predicts same-answer recoverability (beta = +0.522 per budget step); higher reasoning effort delays the peak by 2.59 octaves and widens the high-rank interval across all four architectures, while the registered peak-height contrast spans zero. Controls include expert-identity permutation (peak rate 98.5% to 0.0%), temporal shuffling, risk-set analysis, and a frozen-dial transfer to LiveCodeBench. The paper is candid about limits: routing features reach AUC at most 0.542 for correctness selection versus 0.82-0.91 for answer-side baselines and improve no tested voting or efficiency protocol. Main limitations: near-zero incremental value over plain graph density, family-architecture confounding, and a promised code/data release not yet available.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Routing effective rank traces a reproducible low-high-low trajectory over token budget, with a prominent interior maximum in 98.5% of 3,105 model-question cohorts, abolished (0.0%) by permuting expert identities across rollouts.
- The exact identity log d_eff = h(m) + (1-m) log d_perp attributes about two thirds of the trajectory's log-motion to common-mode mass and about one quarter to residual dimensionality.
- Among non-unanimous cohorts, stepwise increases in common-mode mass strongly predict same-answer recoverability (beta = +0.522 [+0.430, +0.609]), while the residual partial is small and negative.
- Higher reasoning effort delays the d_eff maximum by 2.59 octaves and widens the high-rank interval across all four architectures, whereas the registered peak-height contrast spans zero.
- Routing spectra serve as structural monitors, not answer selectors: no tested voting, early-stopping, or efficiency protocol improves with d_eff, and routing AUCs stay at or below 0.542 versus 0.82-0.91 for answer-side features.
Strengths
Exemplary statistical protocol
Two-way clustered bootstraps over questions x configurations (B=2000), with the prominent-peak interval widening 3.6x vs iid (Section A); a pre-registered Bonferroni family of three hypotheses whose outcomes (including H3's placebo-reinterpreted result) are all reported, per-configuration sign counts (k/10), declared MDE80s, and explicit placebo decision rules (Sections F-G).
Deep, falsifying controls
Expert-identity permutation preserving sparsity/load exactly collapses the prominent-peak rate 98.5%->0.0% and cuts prominence 178x (Section D); frame-order shuffle drops it to 56.2%; risk-set live-only analysis retains 99.8% of cells with peak correlation 0.63; answer-token masking attributes only ~1/5 of collapse depth to answer characters while masking the last 32 pre-answer tokens removes the arc (34.4%).
Honest reporting of nulls and negative utility
Held-out transfer and exactness checks
Weaknesses
Little predictive value beyond plain graph density
Family-architecture confounding with few architectures
Small cluster counts for pooled contrasts
Domain-bounded behavioral conclusions and unreleased assets
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how the internal organization of a test-time reasoning cohort — 64 rollouts of one question under one MoE configuration — reorganizes as generation proceeds, measured without labels through expert-routing traces. It defines d_eff as the entropy-effective dimensionality of a weighted-Jaccard routing-similarity graph (Eq. 1-2) and claims: (1) a reproducible low-high-low trajectory, with a prominent interior maximum in 98.5% of 3,105 cohorts (95% CI [96.7, 99.8], two-way clustered, Section 4.2); (2) an exact decomposition log d_eff = h(m) + (1-m)log d_perp attributing ~0.67 of the trajectory's absolute log-motion to common-mode mass and ~0.24 to residual dimensionality (Section 4.3); (3) behavioral localization — common-mode mass predicts same-answer recoverability (beta = +0.522 on 28,622 steps) and effort delays the peak by +2.59 octaves with wider high-rank occupancy, while the registered peak-height contrast spans zero (Section 4.4, Table A10); (4) a deliberately modest positioning of d_eff as a monitor, not a selector.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is routing effective rank (d_eff) in MoE reasoning?
It is an entropy-based effective dimensionality of a graph whose 64 nodes are reasoning rollouts of the same question and whose edge weights are Jaccard similarities of the models' expert-routing histograms. Low d_eff means routing similarity concentrates in a few spectral directions; high d_eff means it spreads out. The paper shows it rises and then falls as generation proceeds, peaking mid-generation.
Can d_eff predict whether the model's answer is correct?
No. The paper's own AUC analysis shows routing features reach at most 0.542 for correctness selection, while answer-side signals (vote margin 0.833, question prior 0.909) are far stronger, and adding d_eff to voting or early-stopping protocols improved none of them. The authors position d_eff as a structural monitor rather than a selector.
Which models and datasets were used?
Ten configurations of four open-weight MoE models: gpt-oss-20B and gpt-oss-120B at low/medium/high reasoning effort, plus Qwen3-30B-A3B and Qwen3-Next-80B-A3B in Instruct/Thinking variants. Benchmarks are AIME'24, AIME'25, BRUMO'25, HMMT'25, and GPQA, with LiveCodeBench-v5 held out for a final frozen-dial transfer test.
Is the code or data available?
The paper states that activation caches, cohort-level tables, and the complete analysis code will be released under a research-permissive license with pinned software versions; as of this review they were not yet available, so independent verification currently requires regenerating rollouts on the open-weight models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked sampling gaps; content falling inside them (e.g., parts of the method derivation in Section 3 and appendix Sections B-C) was not reviewed, and no completeness judgment is made about it.
- Reference list entries [1]-[4] fell inside a sampling gap, so the citation details for benchmark [3] (BRUMO'25) could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.