CoolMomentum: a method for stochastic optimization by Langevin dynamics with simulated annealing

Area of Science:

• Computational Mathematics
• Machine Learning
• Statistical Physics

Background:

• Deep learning necessitates global optimization of non-convex functions with multiple local minima.
• This challenge mirrors problems in physical simulations, often addressed by Langevin dynamics with Simulated Annealing.

Purpose of the Study:

• To bridge insights from physical simulations to non-convex stochastic optimization in machine learning.
• To propose a novel optimization method, CoolMomentum, inspired by physical principles.

Main Methods:

• Discretizing the Langevin equation to derive a coordinate updating rule.
• Establishing an equivalence between decreasing the momentum coefficient and Simulated Annealing (slow cooling).
• Implementing the proposed CoolMomentum optimization algorithm.

Main Results:

• The coordinate updating rule derived from the Langevin equation is equivalent to the Momentum optimization algorithm.
• A gradual decrease of the momentum coefficient is shown to be analogous to Simulated Annealing.
• CoolMomentum achieves high accuracy on Resnet-20 (Cifar-10) and Efficientnet-B0 (ImageNet).

Conclusions:

• The analogy between physical simulations and machine learning optimization offers valuable insights.
• CoolMomentum presents a novel and effective approach for stochastic optimization in deep learning.
• This method demonstrates strong performance on complex deep learning models and datasets.