XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression
Jundong Hu, Shekar Ramachandran
cs.LG · cs.CL
Quick verdict
Select-then-refit layer merging for LLM depth compression; broad 7-model results, but single seed and only 3/7 CORE margins statistically separable.
Summary
XMerge is a post-training depth-compression method for decoder-only LLMs that removes whole transformer blocks while keeping a standard serving architecture. It decomposes compression into selection and merging: a parameter-free cross-axis selector scores each block on relative hidden-state magnitude change and angular (Block-Influence) change and removes the block quiet on both axes; a reconstructive operator then re-fits the surviving neighboring block (all parameters, 300 Adam steps, lr 1e-5) to reproduce the removed pair's output on 128 WikiText-2 sequences, adding no new inference-time parameters. Across seven Llama/Qwen backbones (0.5B–8B), five baselines, and k∈{1,2,4}, XMerge is strongest at the most aggressive setting: at k=4 it ranks first on six of seven backbones on CORE and six of seven on MMLU, avoiding the 10²–10⁴ perplexity blow-ups shown by four baselines. Task-level bootstraps show only the three largest CORE margins separate from noise (sign test p=0.06); it is the only operator with zero collapses across 14 model-regime cells, and the best-calibrated in a single-backbone ECE probe. Main limitations: a single compression seed with unmeasured reconstruction-seed variance; the closest related operators (ReplaceMe, LLM-Streamline) positioned but not run; WikiText-2 perplexity partially in-domain; and the three headline metrics partly measuring proximity to the reconstruction objective itself.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Re-fitting one surviving standard block to the removed pair's output (300-step, label-free local reconstruction) — not selection — provides most of XMerge's gain over drop and analytic-merge baselines.
- At the most aggressive level (k=4), XMerge ranks first on six of seven backbones on CORE and six of seven on MMLU, while avoiding the 10²–10⁴ perplexity blow-ups of four of five baselines.
- XMerge is the only evaluated operator with zero CORE collapses across all 14 (model, regime) cells at k=4, ranking top-2 in both zero-shot and ICL regimes.
- Cross-axis max-fusion of relative-magnitude and angular scores matters only when the two selection axes disagree, preventing the collapses that BI-only or RM-only selection causes on Qwen3 backbones.
- The one-time reconstruction cost (minutes to 4.4 hours) amortizes via per-token decode savings after roughly 1.9k–24k requests of 256 tokens.
Strengths
Complete evaluation grid with unusually honest statistics
All 126 cells (7 models × 6 methods × k∈{1,2,4}) are reported (§B–§E); §5/Table 10 explicitly marks 4 of 7 k=4 CORE margins as ties, reports the sign test p=0.06 as not significant, and states the bootstrap bounds task noise only, not seed variance.
Transparent baseline auditing
§G discloses LaCo ≡ SWM byte-identical at k=1 on Llama-3-8B, the MKA port correction (argmax-NPIB → official top-down rule), CoMe's KV-head degeneracy on Qwen2.5-0.5B, and SWM's non-comparable native instruction-tuning recovery (§I).
Component analysis cleanly separates operator from selector
Deployment-cost accounting instead of free-lunch framing
Candid limitations section
Weaknesses
Closest competitors positioned but never run
Single seed; thin statistical margins at the headline setting
Advantage concentrated where baselines collapse
Ablations cover a fraction of the grid
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
XMerge (arXiv:2609.02083, PayPal AI) targets post-training depth compression of dense decoder-only LLMs: remove whole transformer blocks, keep a standard L−k architecture. The paper decomposes any such method into selector S and operator M and contributes (i) a parameter-free cross-axis selector combining relative-magnitude (RM) and angular Block-Influence (BI) via s = max(z_RM, z_BI), and (ii) a reconstructive depth-collapse operator that re-fits one surviving neighboring block — all of its parameters — for 300 Adam steps (lr 1e-5, batch 16) to reproduce the removed pair's output boundary on 128 WikiText-2 train sequences. Headline claims: at k=4, first on 6/7 backbones on CORE and 6/7 on MMLU; the only operator with zero collapses across 14 (model, regime) cells; best-calibrated on a one-backbone ECE probe; construction break-even after ~1.9k–24k requests.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is XMerge and how does it compress an LLM?
XMerge is a post-training depth-compression method that removes whole transformer layers while keeping the standard serving architecture. It first selects the layer whose removal is safest using two signals — how little the layer changes hidden-state magnitude and how little it rotates the representation — then re-fits the surviving neighboring layer for 300 optimization steps so its output matches what the removed pair originally produced. No task labels, no architectural changes, and no new inference-time parameters are required.
How much faster is an XMerge-compressed model?
Speedup depends only on depth, not on the compression method: removing k layers yields roughly L/(L−k)× faster decoding. The paper measures a mean batch-1 decode speedup of 1.04×/1.09×/1.17× at k=1/2/4 across seven models. Quality differences between methods are therefore obtained at identical inference cost, and the one-time reconstruction cost (minutes to 4.4 hours) breaks even after roughly 1.9k–24k requests of 256 tokens each.
Which models and baselines were compared in the XMerge paper?
Seven dense decoder-only backbones — Llama-3.2-1B/3B, Llama-3-8B, Qwen2.5-0.5B, and Qwen3-0.6B/1.7B/8B — against five published methods (ShortGPT, LaCo, MKA, SWM, CoMe) at one, two, and four removed layers. Two closely related methods, ReplaceMe and LLM-Streamline, are discussed for positioning but were not run as baselines.
How reliable are XMerge's reported wins?
Mixed but honestly reported. At the most aggressive setting (four layers removed) XMerge ranks first on six of seven models on both CORE and MMLU, yet the paper's own task-level bootstrap shows only the three largest CORE margins separate from noise, the cross-model win count has a non-significant sign test (p=0.06), and a single compression seed means reconstruction-seed variance is unmeasured. Its clearest, best-supported advantage is zero collapses across 14 model-regime cells plus bounded perplexity.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction renders Figures 1–5 as scattered axis-label text; all numeric claims in this review were verified against tables, not figures.
- The metadata-supplied abstract is cut mid-sentence, but the full abstract appears intact in the extracted body text — no paper content was lost to truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.