Fractional-order gradient descent learning for Elman neural networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Conventional integer-order gradient descent methods face limitations in training Elman neural networks, including susceptibility to local minima and slow convergence.
• Elman neural networks are recurrent neural networks widely used in time-series prediction and system identification.

Purpose of the Study:

• To propose a novel fractional-order gradient descent learning algorithm for Elman networks (FO-Elman) to overcome the limitations of conventional methods.
• To theoretically establish the convergence of the proposed FO-Elman algorithm.
• To demonstrate the effectiveness of FO-Elman through experimental validation.

Main Methods:

• Derivation of fractional-order gradient expressions for each layer of the Elman network.
• Establishment of a complete backpropagation framework for fractional-order gradients.
• Incorporation of historical gradient information using the memory property of fractional calculus into the parameter update rule.

Main Results:

• Theoretical proof of the convergence of the FO-Elman algorithm.
• Experimental results on system identification and time-series prediction tasks show improved optimization performance compared to conventional methods.
• Demonstration of FO-Elman's ability to mitigate local minima and accelerate convergence.

Conclusions:

• The proposed FO-Elman algorithm offers a new theoretical and algorithmic tool for training Elman neural networks.
• Fractional-order calculus provides a mechanism to enhance gradient descent optimization by leveraging historical information.
• FO-Elman presents a promising approach for improving the performance of recurrent neural networks in various applications.