Training recurrent neural networks: why and how? An illustration in dynamical process modeling

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Recurrent neural networks (RNNs) are powerful tools for modeling dynamical systems.
• Gradient-based algorithms are commonly used for training RNNs, offering various algorithmic families.
• Selecting the optimal training algorithm is crucial for effective RNN application in process modeling.

Purpose of the Study:

• To provide a general approach to training recurrent neural networks using gradient-based algorithms.
• To introduce four families of training algorithms for RNNs.
• To demonstrate how noise interference influences the choice of algorithm for process modeling.

Main Methods:

• Summarization of general gradient-based training approaches for RNNs.
• Introduction and categorization of four distinct families of training algorithms.
• Empirical evaluation through three case studies modeling dynamical processes.

Main Results:

• The choice of training algorithm significantly impacts RNN performance in process modeling.
• Inappropriate algorithm selection, particularly in the presence of noise, leads to increased prediction errors.
• The study provides clear evidence of the detrimental effects of suboptimal algorithm choices.

Conclusions:

• The effectiveness of RNNs in process modeling is highly dependent on the selected training algorithm.
• Understanding noise characteristics within a process is essential for selecting the appropriate RNN training method.
• This research highlights the practical implications of algorithm selection for reliable dynamical process modeling.