Dynamically Learning to Integrate in Recurrent Neural Networks

Area of Science:

• Machine Learning
• Computational Neuroscience
• Dynamical Systems Theory

Background:

• Recurrent Neural Networks (RNNs) face fundamental challenges in learning long-term dependencies.
• Existing research explores *why* RNNs struggle with long timescales, but the precise learning dynamics remain unclear.
• Gradient descent is a common training method, yet its dynamics in RNNs learning long timescales are not fully understood.

Purpose of the Study:

• To develop a mathematical theory for the learning dynamics of RNNs when learning long timescales.
• To elucidate the role of eigenvalues in the learning process of RNNs.
• To provide a framework for understanding dynamical learning in both artificial and biological neural systems.

Main Methods:

• Mathematical analysis of linear RNNs trained on white noise integration.
• Derivation of low-dimensional dynamical systems to describe learning dynamics.
• Extension of analysis to RNNs learning damped oscillatory filters.

Main Results:

• Identified a low-dimensional system governing learning dynamics when initial weights are small, tracking a single outlier eigenvalue.
• Demonstrated how this outlier eigenvalue precisely captures the learning of long timescales in white noise integration.
• Derived rich dynamical equations for the evolution of conjugate outlier eigenvalues in oscillatory filter tasks.

Conclusions:

• The study provides a novel mathematical framework for understanding RNN learning dynamics over long timescales.
• The findings offer precise insights into how RNNs learn temporal dependencies, relevant to machine learning algorithms.
• The developed theory has implications for understanding learning mechanisms in neuroscience.