Understanding Short-Range Memory Effects in Deep Neural Networks.

Area of Science:

• Machine Learning
• Deep Learning Theory
• Optimization Algorithms

Background:

• Stochastic gradient descent (SGD) is crucial for deep learning, but its efficacy is not fully understood.
• Current theories often model SGD as discretizing stochastic differential equations (SDEs) driven by Brownian or Lévy motion.
• The role of stochastic gradient noise (SGN) in SGD's success is conventionally attributed to its Gaussian or Lévy stable properties.

Purpose of the Study:

• To challenge the conventional understanding of SGN in deep learning.
• To propose a new model for SGD dynamics based on fractional Brownian motion (FBM).
• To explain SGD's tendency to favor flat minima and improve generalization.

Main Methods:

• Analyzing the short-range correlations within the SGN series.
• Modeling SGD as a discretization of an SDE driven by FBM.
• Deriving the first passage time for FBM-driven SDEs.
• Conducting extensive experiments across diverse models, datasets, and training strategies.

Main Results:

• Stochastic gradient noise (SGN) exhibits short-range correlations, deviating from Gaussian or Lévy stable distributions.
• SGD dynamics are better represented as a discretization of an SDE driven by fractional Brownian motion (FBM).
• A larger Hurst parameter in FBM leads to slower escaping rates, causing SGD to remain longer in flat minima.
• Experimental validation confirms the persistence of short-range memory effects in SGN across various deep learning settings.

Conclusions:

• The efficacy of SGD is better explained by its underlying dynamics driven by FBM, not simple noise.
• The short-range memory in SGN provides a theoretical basis for SGD's preference for flat minima, which correlates with good generalization.
• This research offers a novel perspective on SGD, potentially leading to a deeper understanding and improved optimization techniques in deep learning.