What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity
Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem
cs.CL
Quick verdict
Solid study: pruning erodes grounded specificity before intent and triggers over-refusal; single model family and missing significance tests limit generality.
Summary
The paper asks what capabilities are lost when LLMs are structurally pruned for smart-home tool calling. Four Qwen-family models (dense Transformer Qwen3-4B, dense hybrid Qwen3.5-4B/9B with Gated DeltaNet layers, and MoE Qwen3.6-35B-A3B) are pruned with depth (ShortGPT, Angular), width (FLAP), hybrid (2SSP), and expert (REAP) methods, healed by supervised fine-tuning on 50,001 examples, and evaluated on over 19,500 instances from three datasets (proprietary SHTC, HomeBench, HAR). Beyond aggregate exact-match accuracy, degradation is decomposed into action components (operation, device, argument, value) and five task-complexity categories. Three findings stand out: dense models show narrow safe regions (7 of 8 model-method pairs lose under 0.6% at 10% pruning) followed by sharp cliffs, whereas REAP removes 70% of MoE experts with only a -0.37% change; grounded components (device, value) degrade before schema components (operation, argument), with mean gaps at 50% pruning of 24.2% vs 21.1% and 20.7% vs 14.5%; and aggressive dense pruning induces over-refusal (Qwen3-4B refusal on executable HomeBench requests rises from 7.7% to 80.8% under FLAP at 50%). Limitations: single model family, non-factorial method coverage (FLAP/2SSP only on Qwen3-4B), proprietary data, and no statistical significance testing for headline deltas.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Dense Qwen models exhibit narrow safe pruning regions (at 10%, 7 of 8 model-method combinations lose under 0.6% accuracy) followed by sharp cliffs, while REAP expert pruning tolerates 70% expert removal at -0.37%.
- Pruning degrades grounded specificity (device, value) before schema-level intent (operation, argument); at 50% dense pruning mean losses are 24.2% vs 21.1% and 20.7% vs 14.5%.
- Unpruned baseline task accuracy does not predict pruning sensitivity: Partially Executable requests are most sensitive (~10.3% loss per additional 10% pruning) while Infeasible requests are relatively robust.
- Aggressive dense pruning shifts the decision policy toward systematic over-refusal: Qwen3-4B refusal on executable HomeBench requests rises from 7.7% to 80.8% under FLAP at 50% pruning.
Strengths
Fine-grained degradation framework
Decomposes each action into four components (o, d, p, v; §3.1) and stratifies requests into five complexity categories (§3.2), converting aggregate accuracy drops into diagnoses of what breaks and when — e.g., the specificity-first result (Fig. 5) and category-dependent sensitivity (Fig. 6).
Realistic healing protocol
Every pruned model receives an identical SFT recipe on a balanced 50,001-example mixture (Appendix A: 1 epoch, batch 32, LR 5e-6), so measured degradation reflects post-deployment behavior rather than unhealed artifacts — a control most pruning studies omit.
Deployment-relevant over-refusal finding
Stability analyses beyond means
Honest scoping
Weaknesses
No statistical significance testing
Pruning-vs-healing confound
Dense-vs-MoE comparison is not capability-matched
Asymmetric method coverage drives headline numbers
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a deployment-shaped question: when an LLM driving smart-home tool calls is structurally pruned, what breaks first, and at what ratio? Four Qwen-family models — Qwen3-4B (dense Transformer, 36 layers, GQA), Qwen3.5-4B and Qwen3.5-9B (dense hybrid interleaving Gated DeltaNet linear attention with full attention), and Qwen3.6-35B-A3B (hybrid MoE, ~3B active parameters) — are crossed with depth pruning (ShortGPT, Angular), width pruning (FLAP), hybrid pruning (2SSP), and expert pruning (REAP), subject to architectural compatibility. All pruned models are healed with an identical SFT recipe and evaluated on more than 19,500 instances from three datasets. Central claims: (1) dense models have narrow safe regions followed by cliffs while MoE expert pruning tolerates far more removal; (2) grounded specificity (device d, value v) degrades before schema-level intent (operation o, argument p); (3) baseline task difficulty does not predict pruning sensitivity; (4) aggressive dense pruning induces systematic over-refusal. Claims 1-2 are well supported by the reported numbers; claims 3-4 are supported but carry coverage caveats noted below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What happens to LLM smart-home control accuracy when the model is pruned?
According to this study, mild pruning (around 10%) is safe for dense models — 7 of 8 model-method combinations lose under 0.6% accuracy after fine-tuning recovery — but beyond that dense models degrade sharply. A mixture-of-experts model pruned with REAP loses only 0.37% accuracy even with 70% of its experts removed.
Do pruned LLMs fail in ways that aggregate accuracy benchmarks miss?
Yes. The paper identifies two hidden failure modes: (1) device and value predictions break before operation and argument selection, so a pruned model may know what to do but not where or to what setting; (2) heavily pruned dense models start refusing valid commands — refusal on executable requests reached 80.8% at 50% pruning with FLAP on Qwen3-4B, while the model still correctly rejected infeasible ones.
Which pruning method is safest for smart-home LLMs?
In this study, expert pruning (REAP) on the mixture-of-experts model showed by far the widest safe region. For dense models, all tested methods (ShortGPT, Angular, FLAP, 2SSP) stayed safe only up to roughly 10% pruning, and FLAP caused the most severe over-refusal at 50%.
Is the code or data for this paper available?
The paper does not announce a code release. Two of the three evaluation datasets (HomeBench and Home-Assistant-Requests-V2) are public on Hugging Face, but the largest one (SHTC) is proprietary, and only relative category proportions are reported for it.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear in the extracted text only as axis labels and scattered point values; curve shapes and exact per-ratio values could not be verified visually.
- Table 1's column alignment is ambiguous in extraction; instance counts (17,366 HomeBench, 2,194 HAR) were reconstructed from row sums and may be misassigned.
- Several cited models and references (Qwen3.5, Qwen3.6-35B-A3B, 2026 publications) were taken at face value; their specifications could not be independently checked.
- Appendix A omits pruning-calibration data and the refusal-detection criterion, so parts of the setup description are incomplete by construction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.