Adaptive Search-and-Training for Robust and Efficient Network Pruning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Network pruning and neural architecture search (NAS) are key for optimizing artificial neural networks.
• Current methods often train networks before pruning, which can be inefficient.

Purpose of the Study:

• To propose a joint search-and-training approach for learning compact neural networks directly from scratch.
• To challenge the conventional train-then-prune paradigm.

Main Methods:

• An adaptive search algorithm utilizing filter pruning for cold-start optimization.
• ThreshNet, a reinforcement learning-inspired method for coarse-to-fine filter pruning.
• Knowledge distillation via a teacher-student network for robust pruning.

Main Results:

• Achieved a superior balance between network efficiency and accuracy.
• Demonstrated significant advantages over state-of-the-art pruning methods on CIFAR10, CIFAR100, and ImageNet datasets.
• Validated performance on ResNet and VGGNet architectures.

Conclusions:

• The proposed joint search-and-training method effectively learns compact neural networks.
• Offers flexibility in balancing network efficiency and robustness.
• Provides a novel and advantageous approach to network pruning and optimization.