New role for circuit expansion for learning in neural networks

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Many neural pathways exhibit sparse activity downstream from stimuli.
• The function of these expanded neural structures remains largely unknown.
• Increased network expressivity may offer functional benefits.

Purpose of the Study:

• To investigate how expanding neural network structures impacts generalization performance.
• To determine if sparse network expansion improves learning, particularly with noisy data.
• To explore implications for both artificial and biological neural systems.

Main Methods:

• Utilized a teacher-student perceptron framework with noisy labels.
• Trained a student network with a sparse expansion of its input layer.
• Analyzed network behavior in the mean-field limit for stochastic expansions.

Main Results:

• Sparse input layer expansion increased student network capacity and generalization.
• Performance improved even when the expanded structure was pruned post-learning.
• Stochastic, uncorrelated expansions also enhanced generalization, with error dropping as network size increased.

Conclusions:

• Sparse network expansion is a viable strategy for improving generalization in artificial neural networks.
• Findings suggest potential mechanisms for biological neural computation.
• The addition of slack variables in artificial neural networks is related to this observed effect.