Predictive learning as a network mechanism for extracting low-dimensional latent space representations

Area of Science:

• Computational neuroscience
• Machine learning
• Artificial intelligence

Background:

• Artificial neural networks excel at sequential tasks, attributed to emergent low-dimensional latent structures in their activity.
• Understanding how these representations form is crucial for interpreting neural network function and biological neural systems.

Purpose of the Study:

• Investigate if learning to predict world observations generates representations with accessible low-dimensional latent structure.
• Determine when sensory prediction mechanisms align with extracting underlying latent variables in recurrent neural networks.

Main Methods:

• Trained a recurrent neural network model on predicting sequences of observations.
• Quantified representation dimensionality using nonlinear intrinsic dimensionality measures.
• Assessed latent variable extraction via linear decodability.

Main Results:

• Network dynamics revealed low-dimensional, nonlinearly transformed representations of sensory inputs.
• These representations successfully mapped the latent structure of the sensory environment.
• Mathematical arguments provided insights into the emergence of these predictive representations.

Conclusions:

• Learning to predict sensory data naturally yields representations that capture underlying environmental structure.
• These findings offer a framework for analyzing and interpreting experimental data in neuroscience and AI.
• Sensory prediction is a viable mechanism for generating semantically meaningful neural representations.