Stochastic Gradient Descent Introduces an Effective Landscape-Dependent Regularization Favoring Flat Solutions.

Area of Science:

• Deep Learning
• Optimization Theory
• Machine Learning

Background:

• Generalization is a key challenge in deep learning, often addressed by finding flat minima in the loss landscape.
• Stochastic Gradient Descent (SGD) is empirically linked to finding these flat minima, crucial for good generalization.
• Overparameterization in deep learning leads to numerous low-loss solutions, complicating the search for generalizable models.

Purpose of the Study:

• To elucidate the role of SGD noise in finding flat minima and improving generalization.
• To understand how SGD's inherent noise affects the loss landscape and learning dynamics.
• To provide insights into hyperparameter selection for efficient deep learning without divergence.

Main Methods:

• Constructed a simplified model with a continuous set of degenerate minima.
• Approximated minibatch loss landscapes as random shifts of the overall loss function.
• Performed direct simulations of stochastic learning dynamics and solved the associated Fokker-Planck equation.

Main Results:

• SGD noise introduces an anisotropic effective loss term that decreases with landscape flatness.
• This SGD-induced loss acts as regularization, breaking degeneracy and favoring flatter solutions.
• Increased SGD noise strength leads to flatter overall loss landscapes, improving generalization up to a critical point.

Conclusions:

• SGD noise plays a crucial role in promoting generalization by effectively regularizing the learning process towards flatter minima.
• The findings offer a theoretical understanding of SGD's impact on loss landscape geometry and generalization.
• Results suggest implications for selecting optimal hyperparameters, such as learning rate and batch size, to balance generalization and convergence.