Adding Before Pruning: Sparse Filter Fusion for Deep Convolutional Neural Networks via Auxiliary Attention

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Filter pruning is crucial for optimizing deep neural networks (DNNs), reducing model size and computation.
• Current methods often rely on manual, stochastic criteria and multi-stage pipelines (pruning then fine-tuning).
• These existing approaches can be sensitive to hyperparameters and lack robustness.

Purpose of the Study:

• To propose a novel, one-stage filter pruning method that integrates filter selection and learning.
• To enhance the efficiency and robustness of DNNs by adaptively identifying and removing redundant filters.
• To overcome the limitations of manual constraints and hyperparameter sensitivity in traditional filter pruning.

Main Methods:

• Introduced an attention-based architecture for adaptive filter selection and learning within a unified network.
• Developed the Adding Before Pruning (ABP) method, utilizing binary significance scores in an auxiliary attention layer.
• Designed a specific gradient estimator to ensure effective gradient propagation and convergence.
• Implemented simultaneous pruning and training to automatically eliminate redundancy with recoverability.

Main Results:

• Demonstrated favorable performance against state-of-the-art filter pruning algorithms on CIFAR-10 and ILSVRC-2012 benchmarks.
• Showcased the effectiveness of the attention-based approach in focusing on significant filters.
• Validated the robustness and efficiency gains of the one-stage pruning strategy.

Conclusions:

• The proposed ABP method offers a more robust and efficient alternative to existing filter pruning techniques.
• Integrating filter learning and selection in a single stage via attention mechanisms simplifies the process and improves results.
• This approach effectively reduces DNN complexity while maintaining high performance, paving the way for more efficient deep learning models.