diffGrad: An Optimization Method for Convolutional Neural Networks.

Area of Science:

• Machine Learning
• Deep Learning Optimization

Background:

• Stochastic Gradient Descent (SGD) is fundamental to deep neural networks but uses uniform step sizes.
• Existing adaptive methods like Adam rely on past gradients but ignore local gradient changes.
• Efficient deep network optimization requires adaptive step sizes tailored to individual parameters.

Purpose of the Study:

• Introduce a novel optimizer, diffGrad, that adjusts step sizes based on the difference between current and past gradients.
• Analyze the convergence properties of diffGrad using the regret bound approach.
• Evaluate diffGrad's performance against state-of-the-art optimizers in image categorization tasks.

Main Methods:

• Developed diffGrad, an optimizer adjusting step sizes for faster-changing gradients.
• Performed convergence analysis using the online learning framework's regret bound.
• Conducted image categorization experiments on CIFAR10/CIFAR100 using ResNet-based CNNs.

Main Results:

• diffGrad demonstrated superior performance compared to SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam.
• The optimizer showed consistent performance across different activation functions during CNN training.
• Convergence analysis confirmed the effectiveness of the proposed diffGrad technique.

Conclusions:

• diffGrad offers an effective improvement over existing gradient descent optimization methods.
• The adaptive step-size mechanism based on gradient differences enhances deep learning model training.
• diffGrad provides a robust and high-performing optimization solution for deep neural networks.