Coding schemes in neural networks learning classification tasks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Neural networks excel at learning task-dependent features.
• The precise nature of these emergent representations remains poorly understood.
• Understanding feature learning is key to advancing AI and neuroscience.

Purpose of the Study:

• To investigate how learning shapes representations in wide, fully-connected neural networks.
• To analyze the impact of neuronal nonlinearity on emergent feature representations.
• To explore the Bayesian framework for understanding neural network weight posteriors.

Main Methods:

• Utilized a Bayesian framework to model the posterior distribution of neural network weights.
• Analyzed fully-connected, wide neural networks trained on classification tasks.
• Focused on the feature learning ('non-lazy') regime of network operation.

Main Results:

• Networks acquire strong, data-dependent features (coding schemes) where responses correlate with class membership.
• The type of coding scheme critically depends on the neuronal nonlinearity.
• Linear networks develop analog coding schemes; nonlinear networks exhibit sparse or redundant coding via symmetry breaking.

Conclusions:

• Neuronal nonlinearity is a crucial factor determining the nature of emergent representations in neural networks.
• Network properties like weight scaling and nonlinearity significantly shape learned representations.
• Findings offer insights into the mechanisms of feature learning in artificial and biological neural systems.