A homotopy training algorithm for fully connected neural networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Optimization problems in fully connected neural networks with complex structures are challenging.
• Traditional methods may struggle to find global minima for intricate network architectures.

Purpose of the Study:

• To introduce a novel homotopy training algorithm (HTA) for solving complex neural network optimization problems.
• To enhance the probability of finding global minima and improve solution quality for deep learning models.

Main Methods:

• The HTA dynamically constructs neural networks, starting simple and adaptively adding layers and nodes.
• It navigates a continuous path from a simplified to a fully connected network, simplifying the optimization process.
• The algorithm gradually increases model complexity along this path.

Main Results:

• The HTA successfully solves optimization problems for complex neural network structures.
• Numerical results, including VGG models on CIFAR-10, demonstrate the algorithm's effectiveness.
• On VGG13 with batch normalization, HTA reduced the test error rate by 11.86% compared to traditional methods.

Conclusions:

• The HTA offers a robust approach to training complex neural networks, improving solution accuracy.
• It provides a high probability of achieving global minima by simplifying the optimization landscape.
• The algorithm also facilitates the discovery of optimal neural network structures through adaptive building.