Fading memory as inductive bias in residual recurrent networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Residual connections are known to improve training in feed-forward and recurrent neural networks (RNNs) by mitigating gradient issues.
• The specific influence of residual connections on RNN dynamics and fading memory properties remains largely unexplored.

Purpose of the Study:

• To introduce and analyze weakly coupled residual recurrent networks (WCRNNs) to understand the impact of residual connections on RNN dynamics and memory.
• To investigate how WCRNNs influence network performance, dynamics, and memory properties across benchmark tasks.

Main Methods:

• Developed WCRNNs incorporating residual connections designed to yield well-defined Lyapunov exponents for memory analysis.
• Evaluated WCRNN performance, network dynamics, and memory properties on a suite of benchmark tasks.
• Extended findings to non-linear residuals and proposed a weakly coupled residual initialization for Elman RNNs.

Main Results:

• Demonstrated that distinct forms of residual connections in WCRNNs provide effective inductive biases, enhancing network expressivity.
• Showcased residual connections promoting dynamics near the edge of chaos, enabling networks to utilize data's spectral properties, and inducing heterogeneous memory properties.
• Confirmed the applicability of these findings to non-linear residuals and introduced a novel initialization scheme for Elman RNNs.

Conclusions:

• Residual connections in RNNs, particularly within the WCRNN framework, significantly enhance network expressivity and performance.
• WCRNNs offer a valuable approach for studying and optimizing RNN memory properties and dynamics.
• The proposed weakly coupled residual initialization scheme provides a practical method for improving Elman RNNs.