Efficient neural codes naturally emerge through gradient descent learning.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Computer Vision

Background:

• Human sensory systems exhibit enhanced sensitivity to prevalent environmental features, such as horizontal orientations, over rarer ones.
• Understanding how artificial systems develop similar sensitivities is crucial for advancing AI and understanding biological computation.

Purpose of the Study:

• To investigate whether artificial neural networks (ANNs) trained for object recognition develop sensitivity patterns reflecting environmental feature statistics.
• To mathematically elucidate the mechanisms by which gradient descent learning in ANNs promotes sensitivity to common features.

Main Methods:

• Training ANNs on image datasets to perform object recognition tasks.
• Analyzing the sensitivity patterns of learned representations within the ANNs.
• Developing mathematical models to explain the emergence of feature sensitivity during gradient descent optimization.

Main Results:

• ANNs demonstrated heightened sensitivity to frequently occurring features in image data, analogous to human sensory biases.
• Mathematical analysis confirmed that gradient descent learning inherently favors representations sensitive to common features, a principle of efficient coding.
• This phenomenon was observed irrespective of whether learning employed supervised or unsupervised objectives and with unconstrained coding resources.

Conclusions:

• Efficient coding, characterized by preferential sensitivity to common features, can naturally arise from gradient-based learning algorithms in ANNs.
• These findings suggest a convergence between biological sensory processing and artificial learning mechanisms, driven by statistical regularities in data.
• The study provides a theoretical foundation for understanding how efficient representations are formed during the learning process in neural networks.