From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy
Petro Shulzhenko, Gabriele Spadaro, Enzo Tartaglione
cs.LG
Quick verdict
Learned merging removes padding and kernel-growth limits of depth compression; hardware speed-ups shown but gains are modest and learned baselines untested.
Summary
The paper targets depth compression of neural networks: after redundant activation functions are linearized (replaced by identity), adjacent linear or convolutional layers can in principle be merged. Existing analytical merging fails when the second convolution uses non-zero padding (no closed form exists) and inflates the kernel to kA+kC−1, often cancelling latency gains. The authors instead learn the replacement layer: a Shrunk model replaces each merged block with a single trainable convolution (default 3×3) or depthwise-separable block, fine-tuned while all other layers stay frozen, using a weighted combination of cross-entropy, distillation from the linearized Reference model, and activation-map alignment losses. On CIFAR-10, learned merging with 3×3 kernels approaches the analytical 5×5 solution (91.28 vs 91.61). On ImageNet-1k, ResNet18 improves 63.10→64.20, ResNet50 74.93→75.08, MobileNetV2 and Swin-T stay flat, with latency cuts up to 30 ms on Raspberry Pi 5 and 1.12–1.17× batch-1 speed-ups; the analytical 5×5 merge is sometimes slower than the reference (0.79× on Jetson Nano). Ablations show activation-alignment alone collapses to random guessing. Limitations: only 2–6 blocks merged, gains are modest, ResNet34 drops 5.31 points, results are single-run without variance, and no comparison to other learned-merging methods such as DepthShrinker or LayerMerge.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Learning the merged layer removes the need for an analytical solution, enabling merging of convolutions with non-zero second-layer padding (Section 3).
- Choosing a compact replacement kernel (3×3 or depthwise-separable) avoids the kernel growth of analytical merging and converts linearization into real speed-up where the 5×5 analytical merge yields none or negative gains (Tables 4 and 5).
- The combined loss (cross-entropy + distillation + activation alignment) is necessary: local activation alignment alone collapses to random guessing with 4 merged blocks on CIFAR-10 (Table 6).
- Shrunk models match or slightly exceed the linearized Reference accuracy across four linearization methods and four architectures (Tables 2 and 3), with latency cuts on real embedded hardware.
Strengths
Precisely characterized, real gap with the right control experiment
The paper isolates two concrete blockers (no closed-form merge for non-zero second-conv padding; kernel growth to kA+kC−1). Table 5 is the ideal control: identical merged architecture, 3×3 vs analytical 5×5 — 5×5 is slower than the linearized reference on compute-bound devices (×0.79 on Jetson Nano for ResNet18, ×0.74 for ResNet34), while 3×3 gives ×1.12–1.17 everywhere.
Honest negative results and ablations
Table 6 shows activation-map alignment alone collapses to 10.00% (random guessing on CIFAR-10) with 4 merged blocks; Table 4 shows the analytical-equivalent 5×5 kernel actually loses 3.2 ms on Raspberry Pi 5; the authors disclose hyperparameters were tuned only once on CIFAR-10.
Method-agnostic wrt the linearization algorithm
Real-hardware evaluation on edge platforms
Training efficiency and code release
Weaknesses
No comparison to learned-merging competitors
Single-run reporting, no variance or significance
Modest absolute scale of gains
Unanalyzed scaling with merged-block count
Generality claims exceed what is tested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This workshop paper (ITEM @ ECML PKDD 2026) targets the last mile of depth compression: after redundant activations are linearized (replaced by identity), consecutive linear/convolutional layers can in principle be merged. Two known blockers motivate the work: (1) no closed-form merge exists when the second convolution has non-zero padding; (2) analytical merging inflates the kernel to kA+kC−1, which can erase or invert latency gains. The authors instead learn the merged layer: a Shrunk model replaces each merged block with a single trainable convolution (default 3×3; depthwise-separable for MobileNetV2), fine-tuned against a frozen linearized Reference model with a combined loss — cross-entropy (Eq. 1), temperature-scaled distillation (Eq. 2), and activation-map alignment (Eq. 3), weighted w_CE=0.225, w_D=1.97, w_A=0.14. Four claims: merging independent of the linearization method; near-analytical accuracy with smaller kernels; real-hardware speed-ups; all without kernel growth.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does 'From Deep to Shallow' solve?
Depth compression linearizes redundant activation functions so that adjacent layers can be merged, but existing analytical merging (1) has no closed-form solution when the second convolution uses non-zero padding and (2) inflates the merged kernel size to kA+kC−1, which can erase or reverse latency gains. This paper instead learns the merged layer directly, so merging works for arbitrary padding/stride configurations without kernel growth.
How much faster are the compressed models?
On Raspberry Pi 5, latency drops by up to 30 ms (ResNet50) and 17 ms (ResNet18) on ImageNet; at batch size 1 the learned 3×3 merged models run 1.12–1.17× faster than the linearized references, while the analytical 5×5 merge is sometimes slower than the reference itself (e.g., 0.79× on Jetson Nano for ResNet18).
Does merging layers hurt accuracy?
Mostly no: on ImageNet, ResNet18 rises from 63.10 to 64.20 and ResNet50 from 74.93 to 75.08, while MobileNetV2 and Swin-T stay essentially flat. The exception is ResNet34 with 6 merged blocks, which drops from 73.30 to 67.99. Notably, training the merged layer with local activation alignment alone collapses to random guessing (10% on CIFAR-10), so the combined cross-entropy + distillation + alignment loss is essential.
Is code available for this paper?
Yes. The authors publicly released their code at github.com/ShulzhenkoPetr/deep-to-shallow, and the paper discloses all loss weights and training hyperparameters.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction renders equations and some figure content as raw/garbled text (e.g., Eq. 2's KL terms, Fig. 3 axis values); all table values cited in this review were legible.
- No truncation detected: the extraction ends with the complete conclusion, acknowledgments, and reference list through [32].
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.