Training instabilities favor flatter solutions in gradient descent

Area of Science:

• Machine Learning
• Deep Learning Theory
• Optimization Algorithms

Background:

• Classical gradient descent (GD) stability is defined by the loss Hessian's largest eigenvalue (sharpness).
• High learning rates exceeding this threshold typically cause instability and monotonic loss decrease.
• Modern deep networks often achieve peak performance beyond the classical stability regime.

Purpose of the Study:

• To investigate the role of training instabilities in deep learning.
• To demonstrate how instabilities implicitly guide GD towards better generalization.
• To elucidate the geometric mechanisms behind instability-driven improvements.

Main Methods:

• Analysis of gradient descent dynamics beyond the stability threshold.
• Introduction and application of the Rotational Polarity of Eigenvectors (RPE) concept.
• Theoretical extension to stochastic gradient descent and empirical validation with Adam.

Main Results:

• Training instabilities implicitly favor flatter regions of the loss landscape.
• The Rotational Polarity of Eigenvectors (RPE) phenomenon drives Hessian eigenvectors rotation during instabilities.
• Instability-driven flattening improves generalization and is robust to minibatch noise in stochastic GD.

Conclusions:

• Training instabilities play a constructive role in deep learning, enhancing generalization.
• RPE provides a theoretical explanation for how instabilities lead to flatter minima.
• Restoring instabilities in optimizers like Adam further boosts generalization performance.