Toward Generalized Entropic Sparsification for Convolutional Neural Networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) are often overparametrized, leading to computationally expensive and complex architectures.
• Identifying optimal and minimal CNN architectures is an NP-hard problem due to the vast number of possible network configurations and hyperparameters.

Purpose of the Study:

• To develop a computationally scalable and data-driven method for pruning overparametrized CNNs.
• To reduce network complexity while preserving high accuracy through efficient architecture optimization.

Main Methods:

• A layer-by-layer pruning technique based on entropic relaxation is proposed.
• The method utilizes network entropy minimization as a sparsity constraint on pretrained CNNs.
• A numerically scalable algorithm with sublinear scaling cost is employed.

Main Results:

• Achieved high sparsity levels on benchmark datasets: 55%-84% on MNIST (LeNet) and 73%-89% on CIFAR-10 (VGG-16, ResNet18).
• Maintained minimal loss in accuracy across experiments, ranging from 0.1% to 0.5%.

Conclusions:

• The proposed pruning method effectively reduces CNN complexity and parameter count.
• This data-driven approach offers a scalable solution for creating efficient and accurate deep learning models.