Joint Structure and Parameter Optimization of Multiobjective Sparse Neural Network.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Network pruning is crucial for efficient deep learning models.
• Current methods often rely on manual parameter selection, leading to suboptimal performance.
• Inefficient architecture search can degrade network performance, particularly in highly sparse models.

Purpose of the Study:

• To propose a novel joint training method for network pruning using a multiobjective optimization model.
• To address the limitations of user-defined sparsity ratios and inefficient architecture search in existing pruning techniques.
• To develop an efficient approach combining backpropagation (BP) training and multiobjective evolutionary algorithms (MOEAs).

Main Methods:

• Developed a multiobjective sparse model for network pruning.
• Combined BP training for rapid convergence with two modified MOEAs for optimal sparse structure discovery and weight refinement.
• Utilized evolutionary computation to identify network architectures with high performance.

Main Results:

• The proposed method effectively obtains a desired Pareto front (PF).
• Achieved superior pruning results compared to state-of-the-art methods, particularly for highly sparse networks.
• Demonstrated the benefits of the joint training approach in optimizing both network structure and weights.

Conclusions:

• The novel joint training method offers a more effective approach to network pruning.
• The integration of BP and MOEAs provides a robust framework for discovering optimal sparse network architectures.
• This method enhances network performance and efficiency, especially under high sparsity conditions.