Slimming Neural Networks Using Adaptive Connectivity Scores

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Deep neural network (DNN) pruning is crucial for model efficiency.
• Existing methods often involve trial-and-error and hyper-parameter tuning.
• Common pruning approaches include weight-based deterministic constraints and probabilistic frameworks.

Purpose of the Study:

• To develop a single-shot, fully automated DNN pruning algorithm.
• To combine probabilistic and weight-based pruning strategies to overcome limitations.
• To introduce a novel algorithm, slimming neural networks using adaptive connectivity scores (SNACS).

Main Methods:

• Proposed SNACS algorithm combining probabilistic framework with weight matrix constraints.
• Utilized a novel connectivity measure based on adaptive conditional mutual information (ACMI) estimator.
• Introduced operating constraints for automatic pruning percentage determination and a sensitivity criterion for critical filters.

Main Results:

• SNACS demonstrated significant speed improvements, over 17x faster than comparable methods.
• Achieved state-of-the-art single-shot pruning performance.
• Validated on CIFAR10-VGG16, CIFAR10-ResNet56, CIFAR10-MobileNetv2, and ILSVRC2012-ResNet50 benchmarks.

Conclusions:

• SNACS offers an efficient and automated solution for DNN pruning.
• The novel ACMI estimator and operating constraints enable effective and precise pruning.
• SNACS represents a significant advancement in single-shot DNN pruning techniques.