Towards Understanding Convergence and Generalization of AdamW

Area of Science:

• Machine Learning
• Optimization Algorithms
• Deep Learning Theory

Background:

• AdamW introduces decoupled weight decay, differing from L2-regularization's impact on gradient moments.
• The convergence and generalization advantages of AdamW over Adam and L2-Adam require theoretical justification despite its practical success.

Purpose of the Study:

• To theoretically prove the convergence of AdamW.
• To justify the generalization advantages of AdamW compared to Adam and L2-Adam.
• To analyze the stochastic gradient complexity of AdamW for finding stationary points.

Main Methods:

• Convergence analysis of AdamW under general non-convex and PL-conditioned settings.
• Derivation of stochastic gradient complexity bounds for AdamW.
• Bayesian posterior analysis to compare generalization errors.

Main Results:

• AdamW provably converges by minimizing a dynamically regularized loss.
• Established improved stochastic gradient complexity for AdamW, Adam, and L2-Adam, particularly for over-parameterized networks.
• Demonstrated that AdamW achieves smaller generalization errors than Adam and L2-Adam from a Bayesian perspective.

Conclusions:

• AdamW's decoupled weight decay offers provable convergence and superior generalization.
• The theoretical framework clarifies AdamW's distinct optimization dynamics and benefits.
• Experimental validation supports the theoretical findings on AdamW's performance advantages.