A hybrid adaptive preconditioned gradient method with momentum for deep learning.

Area of Science:

• Machine Learning
• Deep Learning Optimization

Background:

• Deep neural networks commonly use first-order stochastic optimization.
• Second-order methods offer faster convergence but face computational overhead in deep learning.

Purpose of the Study:

• Introduce AdapGradm, a novel second-order adaptive optimizer for deep learning.
• Propose HAdapGradm, a hybrid optimizer for seamless transition between AdapGradm and SGD.
• Evaluate the performance and convergence of these new optimizers.

Main Methods:

• AdapGradm uses a diagonal approximate Hessian from first-order derivatives for efficiency.
• HAdapGradm combines AdapGradm with SGD for flexible optimization.
• Convergence is rigorously established under mild conditions.

Main Results:

• AdapGradm achieves computational efficiency comparable to first-order optimizers like Adam.
• HAdapGradm demonstrated lower training errors compared to Adam and baseline optimizers.
• HAdapGradm exhibited superior generalization capabilities in image classification and NLP tasks.

Conclusions:

• AdapGradm and HAdapGradm offer efficient second-order optimization for deep learning.
• HAdapGradm provides a practical and effective alternative to existing optimizers.
• The proposed methods advance deep learning training efficiency and performance.