Gradient calculations for dynamic recurrent neural networks: a survey.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Recurrent neural networks (RNNs) are crucial for processing sequential data.
• Various learning algorithms exist for RNNs, often lacking a unified framework.

Purpose of the Study:

• To present a common framework for diverse RNN learning algorithms.
• To analyze and compare fixed-point and non-fixed-point learning techniques.
• To discuss practical aspects of training and simulating continuous-time RNNs.

Main Methods:

• Unified presentation of learning algorithms including recurrent backpropagation, deterministic Boltzmann machines, backpropagation through time, Elman's history cutoff, and Jordan's output feedback.
• Discussion of forward propagation using adjoint equations.
• Comparative analysis of temporally continuous versus clocked neural networks.

Main Results:

• A unified framework reveals generalizations across different RNN learning algorithms.
• Advantages and disadvantages of continuous-time RNNs are elucidated.
• Practical training "tricks of the trade" for continuous-time and recurrent networks are provided.

Conclusions:

• The unified framework enhances understanding and potential generalization of RNN learning.
• Practical considerations for continuous-time RNNs are addressed, aiding implementation.
• Analysis of computational complexity and learning speed offers guidance for algorithm selection.