A normalized adaptive training of recurrent neural networks with augmented error gradient

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Training recurrent neural networks (RNNs) presents a challenge in balancing convergence speed and training error.
• Existing methods often face trade-offs between transient and steady-state responses.

Purpose of the Study:

• To propose a novel training algorithm, Normalized Adaptive Recurrent Learning (NARL), for RNNs.
• To achieve a better tradeoff between convergence speed and training error.
• To enhance training stability and robustness in RNNs.

Main Methods:

• Introduced an augmented term to the error gradient to precisely model the derivative of the cost function with respect to hidden layer weights.
• Incorporated the influence of activation function gain on training stability.
• Employed an adaptive learning rate to boost the robustness of gradient-based training.

Main Results:

• Computer simulations demonstrated the effectiveness of NARL in a model prediction task.
• NARL showed improved performance compared to conventional Normalized Real-Time Recurrent Learning (N-RTRL).

Conclusions:

• NARL offers an effective solution for balancing conflicting performance metrics in RNN training.
• The proposed method enhances the stability and robustness of RNN training algorithms.
• NARL shows promise for improving the accuracy and efficiency of predictive models.