Two-Phase Switching Optimization Strategy in Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural network optimization is hindered by the vanishing gradient problem and extensive hyperparameter tuning.
• Existing optimization methods often struggle with efficiency and convergence in complex network architectures.

Purpose of the Study:

• To introduce a novel optimization algorithm, stochastic diagonal approximate greatest descent (SDAGD), designed to overcome challenges in deep neural network training.
• To enhance learning efficiency and reduce misclassification rates through a unique two-phase optimization strategy.

Main Methods:

• The stochastic diagonal approximate greatest descent (SDAGD) algorithm employs a two-phase switching optimization strategy inspired by multistage decision control.
• Phase-I involves calculating the greatest step length for rapid descent, while Phase-II switches to an approximate Newton method for accelerated convergence.
• The algorithm utilizes a diagonal approximated Hessian for efficient weight updates and controls step length based on long-term optimal trajectory.

Main Results:

• SDAGD demonstrates steeper learning curves compared to existing optimization techniques.
• The proposed optimizer achieves significantly lower misclassification rates in experimental evaluations.
• The study investigates the effectiveness of SDAGD in deeper networks, analyzing its performance concerning the vanishing gradient problem.

Conclusions:

• The stochastic diagonal approximate greatest descent (SDAGD) algorithm offers an effective solution for optimizing deep neural networks.
• SDAGD provides a robust and efficient method for improving model accuracy and training speed.
• The two-phase strategy and diagonal Hessian approximation contribute to overcoming common optimization challenges in deep learning.