Incremental PID Controller-Based Learning Rate Scheduler for Stochastic Gradient Descent.

Area of Science:

• Machine Learning
• Deep Neural Networks
• Control Theory

Background:

• The learning rate is crucial for effective deep neural network (DNN) training.
• Existing learning rate schedulers often require manual tuning and can be sensitive to initial parameters.

Purpose of the Study:

• To introduce a novel learning rate scheduler for stochastic gradient descent (SGD) based on incremental proportional-integral-derivative (PID) control.
• To reduce the dependence on the initial learning rate and enhance DNN training accuracy.

Main Methods:

• Utilized an incremental PID controller, commonly applied in automatic control, as a learning rate scheduler for SGD.
• Implemented feedback control to dynamically adjust the learning rate based on the relationship between training losses and learning rates, creating PID-Base and PID-Warmup schedulers.

Main Results:

• The proposed incremental PID learning rates achieved higher accuracy compared to multistep learning rates (MSLR), cyclical learning rates (CLR), and SGD with warm restarts (SGDR).
• Demonstrated superior performance on benchmark datasets including CIFAR-10, CIFAR-100, and Tiny-ImageNet-200.
• Showcased reduced sensitivity to the initial learning rate.

Conclusions:

• Incremental PID learning rates offer a robust and effective method for optimizing DNN training.
• This feedback control-based approach has the potential to significantly improve the performance of SGD.