Spring-Block Theory of Feature Learning in Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Deep neural networks (DNNs) exhibit feature learning by progressively reducing data dimensionality.
• Understanding the emergence of this low-dimensional geometry from microscopic dynamics remains a challenge for current theories.

Purpose of the Study:

• To elucidate the role of nonlinearity and noise in feature learning within DNNs.
• To develop a theoretical framework explaining how feature learning progresses across network layers.

Main Methods:

• Constructed a noise-nonlinearity phase diagram for DNNs.
• Developed a macroscopic mechanical theory to model feature learning dynamics.

Main Results:

• Identified distinct regimes where shallow or deep layers learn more effectively based on noise and nonlinearity.
• The proposed mechanical theory successfully reproduces the observed phase diagram.
• Established a link between feature learning across layers and model generalization.

Conclusions:

• Nonlinearity and noise critically influence the effectiveness of feature learning in DNNs.
• A macroscopic mechanical perspective provides a unifying theory for understanding feature learning and generalization in deep networks.