Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks.

Area of Science:

• Computational neuroscience
• Machine learning
• Neural networks

Background:

• Recurrent neural networks (RNNs) are crucial for processing sequential data.
• Understanding how RNNs learn and represent information is key to advancing AI.
• Temporal signal integration is a fundamental task in both artificial and biological systems.

Purpose of the Study:

• To investigate the learning dynamics and emergent representations in RNNs.
• To determine the conditions under which RNNs can integrate multiple temporal signals.
• To analyze the relationship between network architecture, activation functions, and representational capacity.

Main Methods:

• Analytical and numerical investigations of RNNs.
• Characterization of learning dynamics for linear, ReLU, and sigmoidal neurons.
• Analysis of the dimensionality and geometry of the RNN's internal state space.

Main Results:

• RNNs can learn to integrate D scalar signals using n neurons, where D is much smaller than n.
• The internal state of the RNNs evolves on a D-dimensional manifold.
• The shape of this manifold is influenced by the neuron activation function.
• Individual neurons exhibit mixed selectivity, carrying information about all integrated signals.

Conclusions:

• RNNs develop efficient representations for temporal signal integration.
• The emergent manifold structure provides a framework for understanding RNN computations.
• These findings offer insights into biological neural computations, particularly mixed selectivity in the cortex.