GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis
cs.LG
Quick verdict
Training-free cross-layer shared-dictionary compression, claimed SOTA; strong results, but baseline tuning and sub-baseline perplexities merit scrutiny.
Summary
GeoPair is a training-free post-training compression framework for transformers that exploits cross-layer redundancy through shared-dictionary factorization. It contributes three coupled components: (1) a closed-form dictionary update for two layers with distinct Cholesky whitening transforms, derived as a two-term generalized Sylvester equation and solved by simultaneous diagonalization; (2) global layer pairing formulated as maximum-weight matching (Edmonds' Blossom) over a whitened, shift-searched Frobenius column-window distance, replacing Basis Sharing's fixed adjacency; (3) sparse coefficient learning via Hard Thresholding Pursuit with a conjugate-gradient restricted projection. Ablations on Llama3.2 1B and Llama3 8B (Table 1) attribute gains to each component; at 20% compression the full method recovers roughly 94–97% of baseline zero-shot accuracy. On Llama2 7B at 20% compression it reports the best average accuracy among compared pruning methods (63.58 vs LoRAShear 62.22, Table 2) without fine-tuning, outperforms dictionary-learning baselines on Llama3 8B across compression ratios (Fig. 2, Table 6), and generalizes to Qwen3, Gemma3, and Phi-4 up to 32B (Table 3) plus video (Wan2.2) and audio (VibeVoice) generation. Main limitations: sharing is pairwise-only; compressed perplexity falls below the uncompressed baseline on Gemma3 12B and Qwen3 32B without adequate explanation; the reproduced Basis Sharing baseline looks anomalously weak (928 perplexity on Llama3.2 1B); main tables are single runs and no public code link is given.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Shared-dictionary updates for two layers with distinct whitening transforms admit an exact closed-form solution via a two-term generalized Sylvester equation, eliminating heuristic covariance averaging.
- Formulating cross-layer grouping as maximum-weight matching (Edmonds' Blossom) over a whitened structural distance outperforms fixed adjacent pairing.
- The full pipeline with HTP sparsification recovers over 90% of baseline zero-shot accuracy at 20% compression without any fine-tuning, beating dense low-rank and dictionary-learning baselines.
- The method generalizes training-free across architectures (1B-32B), scales, and modalities (text, video, audio).
Strengths
Exact closed-form dictionary update
Two-term generalized Sylvester equation under per-layer whitening solved by simultaneous diagonalization (Sec. 3.2, App. A.1); denominator 1+λiσj ≥ 1 guarantees conditioning, directly replacing Basis Sharing's covariance-averaging heuristic.
Globally optimal grouping with an efficiently computed metric
Maximum-weight matching via Blossom (Sec. 3.4) over a whitened column-window distance computed in O(d·di·(dj−di)) using prefix sums plus 1D cross-correlation (App. B); Table 4 shows +2.3 avg accuracy over greedy adjacency (45.61 vs 43.35, CR=0.4, Llama3.2 1B).
Unusually complete ablation program
Large, consistent component-wise gains
Broad multi-modal, multi-scale validation
Weaknesses
Anomalously weak key baseline
Sub-baseline perplexity left unexplained
Surrogate grouping metric lacks theoretical grounding
Thin statistical reporting
Pairwise-only ceiling with forced pairing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GeoPair (arXiv 2609.25963v1; MWS AI / ITMO University) is a training-free, post-training compression framework for transformer linear layers built on calibration-based activation reconstruction. It claims four contributions: (i) an exact closed-form shared-dictionary update for two layers under distinct Cholesky whitening transforms, posed as a two-term generalized Sylvester equation and solved via simultaneous diagonalization; (ii) globally optimal cross-layer pairing via maximum-weight matching (Edmonds' Blossom) over a whitened column-window Frobenius distance (Eq. 7); (iii) l0-sparse coefficients via Hard Thresholding Pursuit with conjugate-gradient restricted projection; (iv) state-of-the-art accuracy across text, video, and audio at 1B-32B scale. Claims (i)-(iii) are well supported; (iv) is supported within the evaluated comparisons but qualified by baseline-integrity and evaluation anomalies discussed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GeoPair and is it really training-free?
GeoPair is a training-free post-training compression method for transformers. It pairs layers via optimal graph matching (Edmonds' Blossom algorithm), learns one shared dictionary per pair using a closed-form generalized Sylvester equation that respects each layer's activation whitening, and sparsifies coefficients with Hard Thresholding Pursuit — no fine-tuning or backpropagation is involved.
How much does GeoPair compress and what accuracy is retained?
At CR=0.2 (20% compression) on Llama3 8B it keeps 68.17 average zero-shot accuracy vs 70.36 uncompressed (Table 1), with WikiText perplexity 9.43 vs 7.26. Results are reported up to CR=0.4 and up to 32B-scale models (Qwen3 32B).
Which models and modalities was GeoPair evaluated on?
Llama3.2 1B, Llama3 8B, Llama2 7B, Qwen3 8B and 32B, Gemma3 12B, Phi-4 14B for text; the Wan2.2 5B video generator; and the VibeVoice 1.5B audio model. Comparisons include Basis Sharing, SVD-LLM, CoSpaDI, ROCKET, COMPOT, and pruning methods such as LLM-Pruner, WANDA, and LoRAShear.
Does GeoPair need calibration data or GPUs to run?
It needs a small calibration set (256 sequences, FineWeb by default) for the whitening statistics, plus the compression run itself: about 511 s for a 1B model and about 19,900 s (5.5 h) for a 32B model on 4x A100 GPUs (Table 14). No post-compression fine-tuning is used.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are unrecoverable from extracted text; Fig. 2 baseline curves and Fig. 3 video frames could not be inspected (the Fig. 2 x-axis extraction also shows a duplicated 0.4 tick).
- Table 3's layout is ambiguous in the extraction; the Ours-before-Basis-Sharing column mapping was inferred from metric coherence (accuracy and perplexity move inversely across columns).
- Supplementary code was not part of the extracted text; public availability beyond the NeurIPS-checklist claim is unverified.
- arXiv v1 preprint (22 Sep 2026), not yet peer-reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.