Learning Fixed Points of Recurrent Neural Networks by Reparameterizing the Network Model.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Artificial Intelligence

Background:

• Recurrent neural networks (RNNs) are crucial for modeling neural activity and learning in computational neuroscience.
• Fixed points of RNNs are often used to model responses to static stimuli, mirroring visual cortical responses.
• Training RNNs to minimize loss functions evaluated on fixed points is a key challenge, shared with deep equilibrium models in machine learning.

Purpose of the Study:

• To investigate effective methods for training recurrent neural network weights to minimize loss functions evaluated at fixed points.
• To address the limitations of standard Euclidean gradient descent due to singularities in the loss surface.
• To develop alternative learning rules that offer more robust and effective training dynamics.

Main Methods:

• Reparameterization of the recurrent neural network model.
• Derivation of two novel learning rules.
• Analysis of learning dynamics and comparison with standard gradient descent.
• Interpretation of new rules as steepest descent and gradient descent under a non-Euclidean metric.

Main Results:

• Standard gradient descent on the Euclidean space of weights can lead to poor learning performance due to loss surface singularities.
• The derived alternative learning rules effectively avoid these singularities.
• The new learning rules demonstrate significantly more robust and effective learning compared to standard gradient descent.
• The improved learning rules can be understood through the lens of non-Euclidean geometry on the weight space.

Conclusions:

• The common assumption that neural learning follows Euclidean gradients of synaptic weights is questioned.
• Novel learning rules based on non-Euclidean metrics offer superior training for RNNs, particularly for fixed-point computations.
• These findings have implications for both artificial intelligence and understanding biological learning mechanisms.