Improving Deep Neural Networks' Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive...

Area of Science:

• Deep Learning
• Optimization Algorithms
• Machine Learning

Background:

• Momentum is vital for accelerating deep neural network (DNN) training in stochastic gradient-based optimization.
• Current methods often require tedious hyperparameter tuning for momentum, increasing computational burden.

Purpose of the Study:

• To propose a novel adaptive momentum method for DNN training that removes the need for momentum hyperparameter calibration.
• To enhance the efficiency, accuracy, and robustness of DNNs through improved optimization.

Main Methods:

• Developed an adaptive momentum technique inspired by the nonlinear conjugate gradient (NCG) method.
• Integrated this adaptive momentum into Stochastic Gradient Descent (SGD).

Main Results:

• Achieved significant acceleration in DNN training without momentum hyperparameter tuning.
• Demonstrated improved classification accuracy, reducing errors in ResNet110 training on CIFAR10 and CIFAR100.
• Enhanced adversarial robustness of DNNs through improved adversarial training.

Conclusions:

• The proposed adaptive momentum method offers a computationally efficient and effective alternative to traditional momentum techniques in DNN training.
• This method leads to faster training, better performance, and increased robustness, particularly beneficial for adversarial training applications.