Dynamical Channel Pruning by Conditional Accuracy Change for Deep Neural Networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural network (DNN) compression is crucial for efficient deployment.
• Existing channel pruning methods often require repetitive pruning and fine-tuning cycles on pretrained models.
• Indirect criteria are commonly used for pruning, potentially limiting effectiveness.

Purpose of the Study:

• To propose a dynamical channel pruning method that operates early in the training phase.
• To develop criteria directly related to network accuracy for evaluating channel importance.
• To reduce repetitive pruning and fine-tuning processes in DNN compression.

Main Methods:

• A dynamical channel pruning approach is introduced, pruning unimportant channels at the beginning of training.
• Channel importance is evaluated using Conditional Accuracy Changes (CACs), estimated via a channelwise gate that randomly enables/disables channels.
• Two efficient criteria are developed to dynamically estimate CAC during each training iteration.

Main Results:

• The proposed method effectively reduces parameters and computations in various networks (ResNet, VGG, MLP) across datasets (ImageNet, CIFAR, MNIST).
• The method achieves higher or competitive accuracy compared to baseline networks.
• Doubling initial channels and pruning half (DCPH) leads to significant performance improvements by optimizing network structure.

Conclusions:

• Dynamical channel pruning early in training is effective for DNN compression.
• Using accuracy-based criteria like CACs offers a more direct and efficient approach to channel pruning.
• The DCPH strategy shows promise for enhancing network performance through structural optimization.