Anomalous diffusion dynamics of learning in deep neural networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Physics

Background:

• Deep neural network (DNN) learning relies on minimizing non-convex loss functions using stochastic gradient descent (SGD).
• Effective DNNs generalize well by finding solutions in flat minima of the loss landscape.
• The interplay between SGD dynamics and loss landscape geometry is crucial for understanding deep learning.

Purpose of the Study:

• To elucidate how SGD dynamics interact with loss landscape geometry for effective deep learning.
• To investigate the characteristic motion patterns of SGD during the learning process.
• To provide a novel perspective on the efficiency of SGD in DNNs.

Main Methods:

• Analysis of SGD dynamics across various DNN architectures (ResNet, VGG, Vision Transformers).
• Investigation of learning dynamics under different batch sizes and learning rates.
• Adaptation of methods from complex physical systems to study energy landscapes.

Main Results:

• SGD exhibits distinct dynamics: superdiffusion initially, transitioning to subdiffusion as it approaches solutions.
• These dynamics are consistent across different DNNs and training parameters.
• Superdiffusion is linked to fractal-like regions in the loss landscape, facilitating exploration.

Conclusions:

• The fractal geometry of loss landscapes drives superdiffusive SGD behavior, enabling efficient exploration.
• A phenomenological model demonstrates how fractal landscapes guide SGD to flat minima.
• This research offers new insights into SGD effectiveness and implications for designing efficient DNNs.