Optimal design of connectivity in neural network training

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization Theory

Background:

• Supervised training of neural networks (NN) is an optimization process aiming to minimize a cost function.
• The non-convex error surface of NNs often contains local minima (LM) and saddle points, hindering convergence.
• Standard backpropagation (BP) uses local optimization, frequently getting trapped in LM, impeding optimal solutions.

Purpose of the Study:

• To propose a novel stochastic method for global optimization (GO) in neural network training.
• To address the challenge of local minima encountered during supervised NN learning.
• To demonstrate the efficacy of the proposed GO technique for optimal NN training.

Main Methods:

• Development of a stochastic global optimization (GO) method utilizing a uniformly distributed LP tau sequence.
• Testing the developed GO technique on common benchmark problems.
• Application of the GO method to supervised learning tasks for neural networks.

Main Results:

• The proposed stochastic GO method effectively navigates complex error surfaces.
• The technique successfully avoids convergence to local minima (LM).
• Successful optimal supervised training of small-sized neural networks was achieved.

Conclusions:

• The proposed stochastic global optimization method offers a viable solution for overcoming local minima in NN training.
• This approach facilitates more effective and optimal supervised learning for neural networks.
• The method shows promise for improving the training of small-scale neural network models.