Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models.

Area of Science:

• Deep Learning
• Optimization Algorithms
• Machine Learning

Background:

• Deep learning models often require specific optimizers, necessitating extensive trials and inefficient training.
• Existing optimizers may not offer consistent speed improvements across diverse deep network architectures.

Purpose of the Study:

• To develop a novel optimizer, Adan, that enhances deep network training speed and efficiency.
• To provide a consistently performing optimizer across various deep learning tasks.

Main Methods:

• Adan reformulates Nesterov acceleration to create a new Nesterov momentum estimation (NME) method.
• NME is integrated into adaptive gradient algorithms to estimate gradient moments for faster convergence.
• Theoretical analysis shows Adan achieves an ϵ-approximate first-order stationary point with O(ϵ^-3.5) complexity.

Main Results:

• Adan consistently outperforms state-of-the-art (SoTA) optimizers on vision, language, and reinforcement learning tasks.
• Adan achieves new SoTAs on popular networks like ResNet, ViT, GPT-2, and BERT.
• Adan reduces training costs by up to 50% while maintaining or improving performance and demonstrates robustness to large minibatch sizes.

Conclusions:

• Adan offers a significant improvement in deep learning training efficiency and performance.
• The proposed Nesterov momentum estimation method provides a robust and effective approach for adaptive gradient algorithms.
• Adan is a versatile optimizer suitable for a wide range of deep learning applications and architectures.