Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation
Wenhui Chen, Zhifeng Li, Jie Zhou, Navan Preet Singh, Madalina Ciobanu, Chenghua Wang, Qingqing Mao, Ritankar Das
cs.LG · cs.CL
Quick verdict
Diagnoses LRC's stranded MLP capacity (62.5-81.4%); training the full deployed matrix gains +2.36 to +10.45 Avg9 with unusually strong controls; single-seed.
Summary
The paper identifies a deployment–training mismatch in Low-Rank Clone (LRC) distillation: LRC deploys a full-width student MLP but trains only a teacher-column slice, leaving 62.5–81.4% of each deployed matrix's independent linear degrees of freedom unreachable at full inference cost. Under the principle 'train what you deploy', the entire deployed matrix becomes the training object via two mergeable realizations — Dense-LRC (plain dense continuation) and CORE-LRC (zero-initialized teacher-SVD complement path) — both collapsing to one weight of LRC's shape with no added parameters or FLOPs. On matched 10B-token budgets, gains over plain-LRC baselines are +2.36 (Llama3.2-3B→1.5B), +2.71 (Llama3.1-8B→2.7B), and +10.45 Avg9 (Qwen2.5-3B→1.7B, teacher-spectral; the strictly matched dense arm gives +6.39); on Qwen, CORE-LRC at 10B tokens matches the original recipe's ~20B-token accuracy. Controls (basis variants, an equal-parameter slice-confined arm, a cross-recipe 2×2, long-budget saturation, knockout) support attributing the gain to the enlarged reachable set. A 1.5B student matches its ~9T-token teacher's 9-task macro-average within evaluation noise (66.21 vs 66.18) despite a −5.70 MMLU deficit, and a 2.7B student beats Meta's official Llama3.2-3B at ~900× fewer compression tokens (a token count, not a compute claim). All runs are single-seed on one backbone, evaluation is 0-shot MC/QA only, and code is available on request.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LRC's parameterization confines training to the teacher column space, stranding 62.5–81.4% of the deployed MLP matrix's independent linear degrees of freedom at full inference cost with no offsetting benefit (§2, Prop. 1).
- Training the entire deployed matrix from the identical LRC warm start adds +2.36/+2.71/+10.45 Avg9 over matched 10B-token plain-LRC baselines across three teachers, with zero added deployed parameters or FLOPs (Tables 1-2, S1-S2).
- Controls attribute the gain to the enlarged reachable set: an equal-parameter slice-confined arm recovers none of it, basis variants tie within 0.64 Avg9, the gain survives a stripped KL-only recipe, and plain LRC stays ~2.4-2.5 below even at 50B tokens (§6).
- On the widest teacher, CORE-LRC at 10B tokens matches the original recipe's ~20B-token accuracy (63.44 vs 63.43) and the strictly same-lineage dense arm recovers +6.39 (§5.2).
- A 1.5B student matches its teacher's 9-task macro-average within eval noise (66.21 vs 66.18) and a 2.7B student exceeds Meta's official Llama3.2-3B at ~900× fewer compression tokens — explicitly a token count, not a compute claim (§5.2).
Strengths
Crisp formalization of the gap
§2 defines training utilization u and shows LRC's reachable set {T Z⊤} ⊂ R^{dff×r} gives u = 1/ρ, stranding 62.5–81.4% of deployed degrees of freedom; Proposition 1 makes the gap deterministic and distinguishes it from justified gaps (LoRA protects pretrained content; pruning deletes deployed dims), whereas LRC's gap 'buys nothing'.
Control battery well beyond field norm
§6 runs basis controls (dense 65.27 vs spectral 65.40 vs random-completion 64.76, spread 0.64 Avg9), an equal-parameter slice-confined arm (64.02 vs 64.04 plain LRC vs 65.16 full, §6.2), a cross-recipe 2×2 (+1.82 LRC / +2.86 GPD, Table 4), a 50B-token saturation plateau (62.86 vs 65.40, Fig. 3), and a knockout collapsing the student to chance when the 7.4%-energy complement is removed (§6.4).
Honest attribution bookkeeping
Verified baseline reproduction
Zero inference cost by construction
Weaknesses
Single-seed throughout
Equal-parameter control rests on one draw
Basis-conditioning claim is cross-lineage
Narrow evaluation scope with residual MMLU deficit
No public artifact; one backbone
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper diagnoses a parameterization gap in Low-Rank Clone (LRC) distillation (Hao et al. 2025) and repairs it. LRC deploys student MLP weights at the teacher's full intermediate width (dff × d_model(S)) but trains only the teacher-column slice {T Z⊤}, so a fraction f⊥ = 1 − 1/ρ of each deployed matrix's independent linear degrees of freedom — 62.5% at ρ=2.67, 71.4% at 3.50, 81.4% at 5.375 — is unreachable at full inference cost (§2, Prop. 1). Central claims: (1) training the entire deployed matrix from the identical merged warm start adds +2.36/+2.71/+10.45 Avg9 over matched 10B-token plain-LRC baselines on three teachers (Table 1) at zero added inference cost; (2) controls attribute the gain to the enlarged reachable set rather than added parameters, basis choice, or recipe (§6); (3) on Qwen2.5-3B→1.7B, CORE-LRC at 10B tokens matches the original ~20B-token recipe (63.44 vs 63.43), with the strictly same-lineage dense arm at +6.39; (4) a 1.5B student matches its ~9T-token teacher's 9-task macro-average (66.21 vs 66.18) and a 2.7B student beats Meta's official Llama3.2-3B at ~900× fewer compression tokens — carefully labeled a token count, not a compute claim.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'MLP reachability gap' in LRC distillation?
Low-Rank Clone (LRC) deploys a student MLP with the teacher's full intermediate width but trains only the teacher-column slice of each weight matrix. Depending on the teacher's width ratio ρ (2.67-5.375 in this paper), 62.5-81.4% of the deployed matrix's independent degrees of freedom are never trainable, even though they are fully paid for at inference time.
Does training the full matrix cost more at inference?
No. Both proposed realizations (Dense-LRC and CORE-LRC) merge into a single weight matrix of exactly LRC's deployed shape, so deployed parameter count and inference FLOPs are unchanged. Only training-time cost rises: roughly double the trainable parameters, plus ~12% step time for the spectral variant.
How much does the method improve over standard LRC?
On a matched 10-billion-token budget, +2.36 Avg9 for a 1.5B student from Llama3.2-3B, +2.71 for a 2.7B student from Llama3.1-8B, and +10.45 for a 1.7B student from Qwen2.5-3B (on Qwen the strictly matched dense-arm figure is +6.39, and the spectral arm matches the original 20B-token recipe's accuracy at 10B tokens).
Is the 1.5B student really as good as its 3B teacher?
Only on the 9-task macro-average: 66.21 vs 66.18, within evaluation noise. The student exceeds the teacher on several commonsense/QA tasks but trails by 5.70 points on MMLU; the authors present this as an aggregate match, explicitly not a per-task one.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 and S1-S2 appear only as captions in the extracted text; all quantitative statements in this review are drawn from the fully extracted body text and tables (Tables 1-4, S1-S5).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.