On adaptive learning rate that guarantees convergence in feedforward networks.

Area of Science:

• Computational intelligence
• Machine learning algorithms
• Neural network training

Background:

• Feedforward neural networks (FNNs) are widely used but training can be slow and prone to local minima.
• Backpropagation (BP) is a popular training algorithm, but its fixed learning rate can limit performance.
• Lyapunov stability theory provides a framework for analyzing system stability and convergence.

Purpose of the Study:

• To introduce and investigate novel learning algorithms (LF I and LF II) for FNN training based on Lyapunov functions.
• To compare the proposed algorithms with existing methods like BP and Extended Kalman Filtering (EKF).
• To analyze the impact of adaptive learning rates on convergence speed and the ability to avoid local minima.

Main Methods:

• Development of two Lyapunov-based learning algorithms (LF I and LF II) for FNNs.
• LF II is a modification of LF I designed to mitigate local minima issues.
• Performance evaluation on benchmark problems (XOR, 3-bit parity, 8-3 encoder) and a 2D Gabor function, comparing convergence speed and computational time.

Main Results:

• The proposed LF I and LF II algorithms demonstrate significantly faster convergence than BP and EKF for equivalent accuracy.
• LF II shows potential in avoiding local minima, contributing to improved training outcomes.
• The adaptive learning rate, derived from Lyapunov stability theory, is verified as a key factor for accelerated convergence.

Conclusions:

• Lyapunov-based learning algorithms offer a promising alternative for efficient FNN training.
• Adaptive learning rates derived from stability theory enhance convergence speed and generalization.
• The LF algorithms provide a better understanding of FNN learning dynamics, particularly concerning convergence and local minima.