Joint design and compression of convolutional neural networks as a Bi-level optimization problem

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Deep neural networks, particularly convolutional neural networks (CNNs), excel in machine learning and computer vision tasks.
• Designing optimal CNN architectures and hyperparameters presents a significant challenge due to exponential search space growth.
• Existing pruning methods are applied post-design, failing to integrate compression into the architecture development process.

Purpose of the Study:

• To address the challenge of CNN architectural design and hyperparameter selection by integrating filter pruning into the design process.
• To propose a novel bi-level optimization framework for simultaneous architecture generation and filter pruning.
• To introduce and evaluate a new technique, Bi-CNN-D-C, for efficient and accurate CNN development.

Main Methods:

• Framed the problem as a bi-level optimization task: upper level for architecture generation (minimizing network size) and lower level for filter pruning optimization.
• Employed a co-evolutionary migration-based algorithm (CEMBA), a type of evolutionary algorithm, as the search engine for the bi-level optimization.
• Evaluated the Bi-CNN-D-C approach on benchmark image classification datasets: CIFAR-10, CIFAR-100, and ImageNet.

Main Results:

• The Bi-CNN-D-C method demonstrates effective simultaneous optimization of CNN architecture design and filter pruning.
• Comparative experiments show competitive or superior performance against state-of-the-art architectures on benchmark datasets.
• The approach successfully balances network compactness with classification accuracy.

Conclusions:

• Integrating filter pruning during the CNN design process via bi-level optimization is a promising direction for developing efficient deep learning models.
• The proposed Bi-CNN-D-C technique, utilizing CEMBA, offers a robust solution for automated CNN architecture search and compression.
• This method advances the field by providing a more holistic approach to CNN development, leading to potentially lighter and more accurate models.