PID Controller-Based Stochastic Optimization Acceleration for Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization Algorithms

Background:

• Deep neural networks (DNNs) are powerful tools in computer vision and pattern recognition.
• Training DNNs can be computationally intensive and time-consuming.
• Stochastic Gradient Descent with Momentum (SGD-M) is a common optimizer but suffers from issues like overshoot, slowing convergence.

Purpose of the Study:

• To develop a more efficient optimizer for deep neural networks.
• To address the convergence speed limitations of existing optimizers like SGD-M.
• To improve the training process of DNNs by mitigating the overshoot phenomenon.

Main Methods:

• Investigated the relationship between Proportional-Integral-Derivative (PID) control and SGD-M.
• Proposed a novel PID-based optimization algorithm for DNN parameter updates.
• Exploited past, current, and changes in gradients for parameter updates.

Main Results:

• The proposed PID optimizer effectively alleviates the overshoot problem inherent in SGD-M.
• Achieved up to 50% acceleration in training time for popular DNN architectures.
• Maintained competitive accuracy compared to existing methods.
• Demonstrated effectiveness on benchmark datasets like CIFAR10, CIFAR100, Tiny-ImageNet, and PTB for computer vision and natural language processing tasks.

Conclusions:

• The PID-based optimization algorithm offers a significant improvement in DNN training efficiency.
• This method accelerates convergence while preserving model accuracy.
• The approach is validated across diverse computer vision and NLP tasks and datasets.