A Novel Learning Algorithm to Optimize Deep Neural Networks: Evolved Gradient Direction Optimizer (EVGO).

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Gradient-based algorithms are standard for optimizing deep neural network (DNN) parameters.
• The vanishing gradient problem hinders effective parameter optimization in DNNs.
• Existing optimizers like Adam and RMSProp have limitations in addressing this issue.

Purpose of the Study:

• To introduce a novel algorithm, the evolved gradient direction optimizer (EVGO), designed to overcome the vanishing gradient problem in DNNs.
• To evaluate the performance of EVGO against established gradient-based optimization algorithms.
• To analyze the behavior of loss functions during optimization using EVGO.

Main Methods:

• Proposed the evolved gradient direction optimizer (EVGO), which utilizes first-order gradients and a novel hyperplane.
• Compared EVGO with Gradient Descent, RMSProp, Adagrad, Momentum, and Adam.
• Conducted experiments on MNIST, CIFAR-10, and CIFAR-100 datasets using deep convolutional neural networks, AlexNet, and ResNet architectures.

Main Results:

• EVGO demonstrated superior performance compared to all tested gradient-based algorithms across all experimental setups.
• Empirical evaluations confirmed EVGO's effectiveness in handwritten digit recognition and image classification tasks.
• Analysis of loss functions indicated more stable and efficient convergence with EVGO.

Conclusions:

• EVGO is a highly effective algorithm for optimizing deep neural networks, particularly in mitigating the vanishing gradient problem.
• The novel hyperplane introduced by EVGO shows potential as a foundation for developing future optimization algorithms.
• The findings suggest EVGO offers a significant advancement in deep learning optimization techniques.