Neural network surgery: Combining training with topology optimization.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Neural networks excel at complex tasks but require expert knowledge for optimal topology design.
• Current neural architecture search methods are often computationally expensive or rely on trained meta-controllers.
• Automating neural network architecture optimization is crucial for broader accessibility and efficiency.

Purpose of the Study:

• To develop a computationally cheap and effective framework for neural architecture search.
• To fuse neural network training and topology optimization into a single, integrated process.
• To reduce reliance on expert human knowledge and trained parameters in network design.

Main Methods:

• A hybrid genetic algorithm framework was developed for neural architecture search.
• The algorithm integrates topology optimization with network training, allowing structural modifications like adding/removing neuron layers.
• Re-training is applied to adapt network behavior after structural changes, guided by mathematical criteria.

Main Results:

• The algorithm demonstrated significant accuracy improvements (20-40%) on benchmark datasets compared to baseline models.
• It successfully optimized insufficient network topologies, enhancing their learning capabilities.
• Network size was dynamically reduced by up to 15% without compromising achieved accuracy.

Conclusions:

• The developed genetic algorithm offers a computationally efficient alternative for neural architecture search.
• This hybrid approach effectively optimizes neural network performance and resource utilization.
• The method enhances accuracy and rescues underperforming architectures with minimal computational overhead.