LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, Yamuna Prasad
cs.LG · cs.CL
Quick verdict
Calibration-free KS-spectral pruning beats PruneNet/SliceGPT zero-shot; promising, but inconsistent baselines and a weak KS-accuracy correlation temper claims.
Summary
The paper introduces LILA, a calibration-free structured pruning framework for transformer FFN layers. Within a Non-negative Matrix Factorization (NMF) framework, neuron importance is scored by three criteria — Residual Energy, Reconstruction Sensitivity, and the headline KS-Distance Spectrum score, defined as the Kolmogorov–Smirnov distance between the singular-value empirical CDFs of the full and neuron-ablated weight matrix (approximated via a rank-1 downdate) — under three matrix constructions (data-free Abs/Split; activation-weighted Act). Pruning removes the lowest-scored neurons per layer at uniform 20–40% sparsity, preserving the original architecture. Across LLaMA-2-7B, Phi-2, and OPT-1.3B (plus LLaMA-2-13B/OPT-2.7B scaling checks) on five zero-shot benchmarks, unfine-tuned LILA-Spectrum reportedly beats PruneNet's 45M-parameter RL policy by 1.57 pp (LLaMA-2-7B, 25%), WikiText-2-calibrated SliceGPT by up to 6.0 pp, and Wanda by 13.03 pp on OPT-1.3B at 20%; after one epoch of LoRA recovery fine-tuning it lands within 0.48 pp of SliceGPT. Measured speedups reach 1.33× at 25% on the 7B model. An NTK trace-ratio analysis reports 22× lower functional distortion than random pruning — but that figure comes from the Sensitivity variant (Spectrum: 6.4×, OOM-excluded from the NTK evaluation). Adaptive KS-based layer budgets improve perplexity at 25% (11.14→9.50) yet collapse at 30%. Main limitations: inconsistent dense baselines across tables (Phi-2 72.24 vs 70.33; OPT 49.56 vs 53.18), a near-zero KS–accuracy correlation (r=−0.196) presented as confirmatory, no seeds or confidence intervals, and no code availability statement.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The KS distance between the singular-value empirical CDFs of the full and neuron-ablated FFN weight matrix is an effective closed-form, calibration-free neuron importance score (LILA-Spectrum).
- Without fine-tuning, LILA-Spectrum beats PruneNet's 45M-parameter RL policy by 1.57 pp (LLaMA-2-7B, 25% sparsity) and WikiText-2-calibrated SliceGPT by up to 6.0 pp, while preserving the original architecture.
- With one epoch of LoRA recovery fine-tuning, LILA matches SliceGPT to within 0.48 pp on LLaMA-2-7B and Phi-2 using zero calibration data.
- NTK trace-ratio analysis shows a 22× lower functional distortion than random pruning (ρ=2.10 vs 47.38, Phi-2 25%) — attributable to the Sensitivity variant, not the headline Spectrum score.
- KS-based adaptive layer-wise sparsity budgets improve perplexity at 25% (11.14→9.50 on LLaMA-2-7B) but collapse at 30%, revealing single-layer bottlenecks around ~33% layer sparsity.
Strengths
Genuinely data-free core criterion with architecture preservation
Abs/Split variants (Theorem 3.1, Algorithm 1) need no calibration data, no forward passes, and keep dense FFN tensors — unlike SliceGPT's irreversible PCA rotation; the KS score (Definitions 3.2–3.3) is a closed-form rule, with LILA-Sensitivity completing in 2.4 min on LLaMA-2-7B.
Unusually broad empirical grid with honest negative results
9 configurations (3 criteria × 3 variants) × 3 models × 4 sparsities (Table 1), a 48-run rank sweep (Table 15), 13B/2.7B scaling checks (Appendix K); the adaptive-budget collapse at 30% (PPL 281.47) and NTK OOM exclusions are transparently disclosed.
Hardware relevance is actually measured
The recovery phase is properly isolated
Weaknesses
Dense baselines are internally inconsistent across tables
The headline NTK claim belongs to the wrong variant
A disconfirming correlation is spun as confirmation
No variance quantification anywhere
Baseline comparability is fragile
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LILA (arXiv 2609.11163) targets structured FFN pruning of LLaMA-2-7B, Phi-2, and OPT-1.3B (plus LLaMA-2-13B/OPT-2.7B in Appendix K) at uniform 20–40% sparsity, evaluated on five zero-shot benchmarks (PIQA, HellaSwag, ARC-Easy, ARC-Challenge, WinoGrande) and WikiText-2 PPL. Central claims: (1) a closed-form, data-free importance score — the KS distance between singular-value ECDFs of the full and neuron-ablated FFN matrix (Definitions 3.2–3.3); (2) no-fine-tuning superiority over PruneNet (+1.57 pp at 25% on LLaMA-2-7B: 60.20 vs 58.63) and WikiText-2-calibrated SliceGPT (up to +6.0 pp at 30%: 57.50 vs 51.50, Table 9); (3) post-LoRA-RFT parity with SliceGPT within 0.48 pp on LLaMA-2-7B and Phi-2; (4) NTK-grounded theory (22× lower trace ratio than random pruning); (5) adaptive KS-budgeted sparsity with better PPL at 25% (11.14→9.50) but collapse at 30% (281.47). Claims are explicit and mostly testable; claim (4) is the weakest link.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LILA pruning?
LILA is a training-free, calibration-free structured pruning method for the FFN layers of large language models. It ranks each neuron by the Kolmogorov–Smirnov distance between the singular-value distributions of the original FFN weight matrix and the matrix with that neuron removed, then deletes the least important neurons. The pruned model keeps a standard dense architecture, so it runs on normal hardware without special kernels.
How does LILA compare to SliceGPT and PruneNet?
According to the paper, on LLaMA-2-7B at 25% FFN sparsity with no fine-tuning, LILA-Spectrum scores about 1.6 pp higher zero-shot accuracy than PruneNet's 45M-parameter RL policy and about 4.7–6.0 pp higher than SliceGPT calibrated on WikiText-2. After one epoch of LoRA recovery fine-tuning, LILA is reported within 0.48 pp of SliceGPT on LLaMA-2-7B and Phi-2 without using any calibration data, while keeping the original architecture.
Does LILA need calibration data or GPU training?
The core Abs and Split variants need no data and no forward passes — importance scores are computed directly from weight matrices in minutes (2.4–44 min on LLaMA-2-7B depending on configuration). Only the optional Act variant uses 128 calibration samples, and recovery fine-tuning (one epoch of LoRA) is a separate, optional post-processing step.
Does LILA actually make inference faster?
Yes, because it removes whole FFN neurons and leaves dense weight tensors: the paper reports 33.96 vs 25.60 tokens/s (a 1.33× speedup) on LLaMA-2-7B at 25% sparsity and a 1.64× speedup on LLaMA-2-13B, using standard dense matrix-multiplication libraries.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled 40k-character excerpt with explicitly marked [...sampling gap...] sections; content inside those gaps (e.g., the full post-RFT Table 2, appendix sections A and E–G, and the continuation of §3.1) could not be consulted, so judgments relying on such content are limited — hence medium confidence.
- No code or artifact availability statement appears in the provided excerpt; the paper may declare one in an unseen section.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.