Doing the Impossible: Why Neural Networks Can Be Trained at All

Area of Science:

• Machine Learning
• Computational Neuroscience
• Statistical Physics

Background:

• Deep neural networks (DNNs) face training challenges as their size increases, raising questions about data requirements.
• Similar challenges exist in protein folding, spin glasses, and biological neural networks, where complex systems find optimal configurations efficiently.

Purpose of the Study:

• To investigate the mechanism enabling reliable training of DNNs with limited data.
• To elucidate how complex systems find optimal configurations.
• To propose methods for accelerating DNN training by exploiting this mechanism.

Main Methods:

• Utilizing the concept of mutual information between successive layers in DNNs.
• Analyzing the impact of network structure on mutual information.
• Relating high mutual information to a reduced effective number of free parameters.

Main Results:

• Adding structure to DNNs increases mutual information between layers.
• High mutual information implies an exponentially smaller effective parameter count compared to tunable weights.
• This finding provides insight into why DNNs with more weights than training samples can be reliably trained.

Conclusions:

• A mechanism exists that forces complex systems, including DNNs, into low-dimensional manifolds, mitigating the curse of dimensionality.
• Mutual information serves as a key metric to understand and potentially exploit this mechanism.
• Structured deep learning models exhibit higher mutual information, leading to more efficient and reliable training.