Theory of Gating in Recurrent Neural Networks

Area of Science:

• Computational Neuroscience
• Machine Learning Theory

Background:

• Recurrent neural networks (RNNs) are key dynamical models in machine learning (ML) and neuroscience.
• Existing theories primarily address RNNs with additive interactions.
• Gating (multiplicative interactions) is crucial in biological neurons and high-performing ML RNNs.

Purpose of the Study:

• To investigate the impact of gating on the collective dynamics of RNNs.
• To demonstrate how gating controls timescales and dimensionality.
• To explore novel dynamical states and transitions enabled by gating.

Main Methods:

• Theoretical analysis of RNNs incorporating multiplicative gating mechanisms.
• Examination of network dynamics related to timescales and dimensionality control.
• Characterization of emergent states, including marginally stable integration and chaotic transitions.

Main Results:

• Gating enables flexible control over network timescales, leading to a marginally stable integrator state without parameter fine-tuning.
• Gating provides context-dependent memory reset capabilities.
• A novel discontinuous chaotic transition is induced by gating, decoupling topological and dynamical complexity.

Conclusions:

• Gating offers a powerful mechanism for flexible control of RNN dynamics, enhancing integration and memory functions.
• Gating introduces novel dynamical phenomena, such as context-dependent memory reset and unique chaotic transitions.
• Phase diagrams derived from this work offer practical guidance for ML practitioners in parameter initialization.