Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Ryan Pegoud, Manoj Rohit Vemparala, Pierpaolo Mori, Alexander Frickenstein, et al.
cs.CV · cs.AI · cs.LG
Quick verdict
FACTS+CoRS lift Fisher-weighted SVD compression of ViTs without finetuning; broad plausible gains, but thin margins vs Shampoo2 and no variance reporting.
Summary
Motivated by the observation that more accurate Kronecker approximations of the empirical Fisher information matrix do not predict post-compression accuracy in Vision Transformers, the authors propose FACTS, a structured Fisher approximation that enforces token-local aggregation while preserving within-token activation-gradient coupling, derived via a token-wise power iteration under a zero-cross-moment assumption. They also introduce CoRS, a three-stage rank search that profiles per-layer KL-divergence error, densifies profiles with a monotonicity-checked rolling-window cubic interpolation, and solves a MILP for layer-wise ranks under a hard FLOP budget. Without finetuning, FACTS reaches 77.5% Top-1 on DeiT-B and 65.9% on Swin-B at 50% linear-layer FLOPs, beating the strongest prior SVD baselines (75.0%, 60.1%) by 2.5 and 5.8 points; CoRS lifts these to 81.3% and 74.9%. Experiments span DeiT, Swin, ConvNeXt, and MambaVision in S/B/L sizes, plus COCO detection, ADE20k segmentation, hardware throughput benchmarks, and an exploratory Qwen3-1.7B study. Key caveats: at uniform compression the margin over Shampoo2, the closest competitor in the paper's own comparison, is only 0.3-0.7 points; all results are single runs without variance estimates; and several tables report inconsistent baselines or budgets (e.g., 83.3 vs 81.8 for DeiT-B).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Higher cosine similarity of Kronecker-factored FIM approximations does not predict post-compression accuracy for Fisher-weighted SVD compression of ViTs (Sec. 4.1, Fig. 1c, Table 2).
- FACTS, combining token-local aggregation with within-token activation-gradient coupling, improves zero-shot post-compression accuracy over six SVD baselines across four architectures at matched FLOPs without finetuning (Table 1).
- CoRS, a MILP rank search over interpolated KL-divergence error profiles, adds up to +9.0 p.p. on Swin-B at fixed FLOPs and outperforms prior rank searches at lower wall-clock cost (Table 3).
- Compressed models deliver ~1.6x (DeiT-B) / ~1.4x (Swin-B) real throughput gains and match fully retrained pipelines on COCO and ADE20k with zero or one epoch of finetuning (Fig. 2, Tables 6-7).
Strengths
Counterintuitive, well-instrumented motivating analysis
Sec. 4.1 / Fig. 1c / Table 2 decouple FIM approximation fidelity from post-compression accuracy: KFAC-reduce reaches higher cosine similarity than KFAC-expand (0.129 vs 0.119) yet loses 8.9 p.p. (67.0% vs 75.9%), while Shampoo2 (cos-sim 0.17) and FACTS (0.128) land at 77.2% and 77.5%. The two structural axes (token aggregation, activation-gradient coupling) are analyzed separately with dedicated diagnostics (cross-token Gram heatmaps, distance correlation).
Unusually broad, budget-controlled evaluation
Four architectures (DeiT, Swin, ConvNeXt, MambaVision) at S/B/L sizes, three budgets (40/50/60% linear-FLOPs, Tables 1, 8, 9), downstream COCO Mask R-CNN and ADE20k UPerNet (Tables 6-7), throughput on V100/A100/H100/CPU including an honest negative result that 2:4 sparsity is slower than dense on ViT workloads (Fig. 5), and an exploratory Qwen3-1.7B study (Table 11).
Practical, cheap rank search with component ablation
Deployment realism and honest scoping
Weaknesses
Thin margin over the closest competitor, which is absent from the main table
No statistical rigor anywhere
Internal reporting inconsistencies
Test-set contamination in the motivating study
Theory-practice gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets training-free (zero-shot) low-rank compression of Vision Transformers via Fisher-weighted SVD. It makes two contributions: (1) FACTS, a structured Kronecker approximation of the FIM that keeps token-local curvature and within-token activation-gradient coupling while discarding cross-token mixed moments (Assumption 4.1, Theorem 4.2, Algorithms 1-2); (2) CoRS, a three-stage rank allocator - per-layer KL-divergence error profiling, monotonicity-checked rolling-window cubic interpolation, and MILP via CBC/pulp - that minimizes proxy error under a hard FLOP budget (Sec. 3.2, 5.1). Central claims: FIM-approximation fidelity (cosine similarity) is a poor predictor of post-compression accuracy (Sec. 4.1, Fig. 1c, Table 2); FACTS beats six SVD baselines across DeiT/Swin/ConvNeXt/MambaVision at matched FLOPs without finetuning (Table 1); CoRS adds up to +9.0 p.p. on Swin-B (65.9 to 74.9) and outperforms prior rank searches at lower cost (Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FACTS in model compression?
FACTS is a structured Kronecker approximation of the Fisher information matrix for Fisher-weighted SVD compression of Vision Transformers. It aggregates curvature statistics token-locally (discarding cross-token mixed moments) while preserving the dependence between a token's activations and its output gradients, computed by a token-wise power iteration. It is applied layer-wise, replacing each weight matrix with two smaller low-rank layers, without any finetuning.
Does a FACTS-compressed Vision Transformer need finetuning?
No. The paper's main results are zero-shot: at 50% of linear-layer FLOPs, DeiT-B retains 77.5% and Swin-B 65.9% ImageNet Top-1 accuracy. A single finetuning epoch is used only in the structured-pruning comparison (81.4% on DeiT-B) and to close the final gap on downstream COCO/ADE20k tasks.
How much speedup does FACTS compression give?
The 50%-compressed DeiT-B and Swin-B models show roughly 1.6x and 1.4x throughput improvements respectively on V100, A100, and H100 GPUs and an Intel Xeon CPU. FLOPs drop from 33.9 to 17.1 GFLOPs on DeiT-B and from 30.3 to 15.5 GFLOPs on Swin-B.
Does FACTS also work on large language models?
Only an exploratory study on Qwen3-1.7B is reported: WikiText perplexity of 40.8 versus 57.7 for SVD-LLM and 44.6 for Shampoo2, with the best zero-shot accuracy (36.5%). The authors explicitly defer LLM-specific challenges to companion work, so this is indicative rather than a validated LLM result.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked '[...sampling gap...]' regions (part of the Sec. 4 analysis, part of the references, and part of Appendix A.5-B.1); judgments about those specific regions rest on the surrounding visible text only.
- The code repository (github.com/MoritzTho/FACTS) could not be inspected; code availability is taken from the paper's own statement.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.