Adaptive Filter Pruning via Sensitivity Feedback.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) require acceleration for efficient deployment.
• Existing filter pruning methods, often based on L1-regularized training, face challenges with scaling-invariance and selecting optimal penalty coefficients.
• A trade-off exists between the pruning ratio (model compression) and prediction accuracy drop.

Purpose of the Study:

• To introduce a novel, lightweight filter pruning method called Adaptive Sensitivity-Based Pruning (ASTER).
• To address the limitations of existing pruning techniques, specifically scaling-invariance and the difficulty in balancing pruning ratio and accuracy.
• To enable efficient acceleration of DNNs without specialized hardware or libraries.

Main Methods:

• ASTER achieves scaling-invariance by preserving unpruned filter weights.
• It dynamically adjusts the pruning threshold during the training process.
• Sensitivity of the loss to the threshold is computed efficiently using L-BFGS on batch normalization layers, enabling dynamic threshold adaptation.

Main Results:

• Extensive experiments on state-of-the-art CNNs (ResNet-50, MobileNet v2, MobileNet v3-small) demonstrate ASTER's effectiveness.
• ResNet-50 achieved over 76% FLOPs reduction with a 2.0% Top-1 accuracy drop on ILSVRC-2012.
• MobileNet v2 and v3-small showed significant FLOPs reduction (46.6% and 16.1% respectively) with minimal accuracy degradation.

Conclusions:

• ASTER is an effective lightweight pruning method for accelerating DNNs.
• The method successfully balances model compression and accuracy preservation.
• ASTER offers a practical solution for deploying efficient deep learning models on various platforms.