Neural Network Structure Optimization by Simulated Annealing.

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Large neural networks face challenges with over-parameterization, limiting their deployment on resource-constrained edge devices.
• Edge devices require pre-trained, compressed neural networks due to limited computational power and scarce internet resources for training.
• Existing network compression methods often rely on computationally intensive back-propagation for weight training.

Purpose of the Study:

• To optimize neural network structure for effective compression and performance preservation on edge devices.
• To introduce a compression methodology that avoids computationally expensive back-propagation during network pruning.
• To demonstrate the feasibility of aggressive pruning for reducing neural network complexity.

Main Methods:

• Utilized the simulated annealing algorithm for neural network structure optimization and aggressive pruning.
• Focused solely on structural optimization via simulated annealing, excluding back-propagation for weight training.
• Evaluated the performance of pruned networks to ensure significant complexity reduction without performance degradation.

Main Results:

• Simulated annealing effectively reduced the complexity of fully connected neural networks.
• Network performance was maintained despite aggressive pruning and significant complexity reduction.
• The proposed method successfully compressed neural networks without requiring back-propagation for weight training.

Conclusions:

• Simulated annealing offers an efficient heuristic-based approach for neural network compression.
• This method enables the practical deployment of complex neural networks on edge devices with limited resources.
• The approach provides a viable alternative to traditional back-propagation-dependent compression techniques.