SAGRAD: A Program for Neural Network Training with Simulated Annealing and the Conjugate Gradient Method.

Area of Science:

• Computational neuroscience
• Machine learning
• Artificial intelligence

Background:

• Neural networks are powerful tools for classification tasks.
• Traditional training methods can struggle with non-quadratic error surfaces common in neural networks.
• Efficiently navigating complex weight spaces is crucial for effective neural network training.

Purpose of the Study:

• To introduce SAGRAD (Simulated Annealing GRADient), a Fortran 77 program for neural network classification.
• To detail the hybrid training approach combining simulated annealing and Møller's scaled conjugate gradient algorithm.
• To demonstrate the program's effectiveness on sample datasets.

Main Methods:

• Implementation of Møller's scaled conjugate gradient algorithm for efficient gradient computation and Hessian-vector multiplication.
• Integration of simulated annealing for weight re-initialization to escape local minima.
• Strategic use of simulated annealing to overcome plateaus or local minima encountered by the conjugate gradient method.

Main Results:

• SAGRAD effectively trains neural networks for classification using batch learning.
• The hybrid approach successfully navigates complex weight spaces, improving convergence.
• Demonstrated performance on two distinct training datasets showcases the program's utility.

Conclusions:

• SAGRAD provides an effective hybrid training strategy for neural networks.
• The combination of simulated annealing and scaled conjugate gradient offers robustness against local minima.
• The program is a valuable tool for classification tasks in machine learning and AI.