Accelerated optimization in deep learning with a proportional-integral-derivative controller.

Area of Science:

• Deep Learning
• Optimization Algorithms
• Control Theory

Background:

• Understanding deep learning optimization remains challenging due to algorithm instability and poor interpretability.
• Gradient-based optimization can be modeled as continuous-time dynamical systems.
• Feedback control offers a new perspective for developing robust and explainable optimizers.

Purpose of the Study:

• To present a novel optimization framework, the controlled heavy-ball optimizer.
• To develop a deterministic continuous-time optimizer, Proportional-Integral-Derivative Accelerated Optimizer (PIDAO), using a PID controller.
• To provide theoretical convergence analysis for PIDAO in unconstrained optimization.

Main Methods:

• Implementing a Proportional-Integral-Derivative (PID) controller within an optimization framework.
• Developing a deterministic continuous-time optimizer (PIDAO).
• Conducting theoretical convergence analysis for unconstrained (non-)convex optimizations.
• Deriving PIDAO-family schemes for deep neural network training via discretization.

Main Results:

• PIDAO demonstrates a more aggressive capacity for exploring the loss landscape compared to classical optimizers.
• The PID controller in PIDAO leads to lower computational costs.
• Experimental evaluations show PIDAO accelerates convergence and enhances deep learning accuracy.
• PIDAO achieves state-of-the-art performance against advanced algorithms.

Conclusions:

• PIDAO offers a robust, accurate, and explainable approach to deep learning optimization.
• The integration of control theory into optimization dynamics yields significant performance improvements.
• PIDAO represents a promising advancement in high-performance optimization algorithms for deep learning.