Cross-Entropy Pruning for Compressing Convolutional Neural Networks

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep convolutional neural networks (CNNs) achieve high performance but suffer from large model sizes and significant storage requirements.
• Model compression is crucial for deploying CNNs on resource-constrained devices and reducing computational costs.

Purpose of the Study:

• To propose an efficient and robust pruning approach for compressing CNNs.
• To reduce the storage costs and computational complexity of deep CNN models without substantial accuracy degradation.

Main Methods:

• Introduced Cross-Entropy Pruning (CEP), a group-wise connection pruning method based on cross-entropy errors.
• Developed Highest Cross-Entropy Pruning (HCEP) to further enhance accuracy by retaining weights with the highest CEP.
• Validated methods on challenging, low-redundancy networks like LeNet-5 and AlexNet.

Main Results:

• CEP achieved 0.08% accuracy drop on LeNet-5 with only 16% of original parameters for MNIST dataset.
• Reduced AlexNet storage by ~75% on ImageNet (ILSVRC 2012), with minimal top-1 (0.4%) and top-5 (0.2%) error increases.
• CEP and HCEP outperformed existing methods in accuracy and stability on LeNet-5.

Conclusions:

• CEP and HCEP offer effective strategies for compressing CNNs, significantly reducing model size and storage.
• These pruning techniques enable high-performance computation for computer vision tasks like object detection and style transfer.
• The proposed methods provide a practical solution for deploying efficient deep learning models.