Weight Decay With Tailored Adam on Scale-Invariant Weights for Better Generalization

Area of Science:

• Deep Learning
• Machine Learning Optimization

Background:

• Weight decay (WD) is a key regularization technique for improving deep learning model generalization.
• WD's effectiveness is diminished with adaptive optimization algorithms like Adam, unlike its performance with SGD.
• The underlying reasons for WD's deficiency with Adam, particularly concerning weight norm behavior, remain unclear.

Purpose of the Study:

• To investigate why weight decay is less effective with the Adam optimizer compared to SGD.
• To propose a modified Adam optimizer that effectively integrates weight decay for improved generalization.
• To demonstrate the proposed method's ability to find solutions with smaller weight norms and superior generalization performance.

Main Methods:

• Analyzed the weight norm dynamics during training for Adam versus SGD.
• Introduced a regularization term on the adaptive learning rate within Adam to improve WD compatibility.
• Incorporated the first moment of weight decay to further boost regularization effects.

Main Results:

• Demonstrated that Adam's weight norm increases rapidly during training, hindering WD effectiveness.
• The proposed tailored Adam method successfully finds solutions with smaller weight norms.
• Significant improvements in test accuracy were observed compared to standard Adam and even SGD on image classification tasks (e.g., CIFAR-10, CIFAR-100).

Conclusions:

• The proposed modification effectively addresses the incompatibility between weight decay and the Adam optimizer.
• The tailored Adam method offers a practical solution for enhancing deep learning model generalization.
• This approach provides a valuable advancement for optimization techniques in deep learning, particularly for image classification.