How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Recurrent Neural Networks (RNNs) struggle with learning long-term temporal dependencies.
• NARX (Nonlinear Autoregressive with Exogenous Inputs) networks show improved performance due to 'jump-ahead' connections facilitating gradient propagation.
• The hypothesis is that increased memory order, not just architecture, enhances long-term dependency learning.

Purpose of the Study:

• To empirically validate if increasing embedded memory order improves long-term dependency learning in RNNs.
• To investigate this effect across different RNN architectures: globally recurrent, locally recurrent, and NARX networks.
• To provide practical insights for enhancing RNN robustness in sequence learning.

Main Methods:

• Comparative analysis of three RNN architectures (globally recurrent, locally recurrent, NARX).
• Systematic variation of embedded memory orders within each architecture.
• Evaluation on two distinct simple long-term dependency problems.

Main Results:

• All tested RNN architectures demonstrated significant improvements in learning long-term dependencies as embedded memory order increased.
• Increased memory order consistently enhanced performance across globally recurrent, locally recurrent, and NARX networks.
• The findings support the hypothesis that memory order is a key factor for robust long-term dependency learning.

Conclusions:

• Increasing the embedded memory order is an effective strategy to improve RNN performance on long-term dependency tasks.
• This approach offers a general method to enhance the robustness of various RNN architectures without fundamental architectural changes.
• Users can improve their existing RNN models' capability for sequence learning by adjusting the memory order.