Fast Filter Pruning via Coarse-to-Fine Neural Architecture Search and Contrastive Knowledge Transfer.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Filter pruning is crucial for lightweighting convolutional neural networks (CNNs).
• Existing pruning methods, including neural architecture search (NAS)-based ones, incur significant computational costs.
• Lightweighting the pruning process itself is essential for enhancing CNN usability.

Purpose of the Study:

• To develop a computationally efficient filter pruning algorithm for CNNs.
• To improve the performance and convergence speed of pruned networks.
• To reduce the overall computational cost associated with CNN lightweighting.

Main Methods:

• Proposed a coarse-to-fine NAS algorithm combined with filter importance scoring (FIS) for subnetwork candidate selection.
• Developed a fine-tuning structure utilizing contrastive knowledge transfer (CKT) and a memory bank for interim subnetwork information.
• Implemented an efficient search process that avoids the need for a supernet.

Main Results:

• Achieved significant speed efficiency in pruning compared to state-of-the-art (SOTA) methods.
• Demonstrated pruning of ResNet-50 on ImageNet-2012 by 40.01% with no accuracy loss.
• Reduced computational cost to 210 GPU hours, showcasing superior efficiency over existing NAS-based algorithms.

Conclusions:

• The proposed method offers a computationally efficient approach to filter pruning for CNNs.
• The integration of NAS and CKT enables pruned networks to achieve high performance with fast convergence.
• This technique presents a practical solution for lightweighting CNNs with reduced resource demands.