A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions

Area of Science:

• Machine Learning
• Computational Statistics
• Bayesian Inference

Background:

• Bayesian learning in big data presents computational challenges, particularly with multi-modal or non-convex distributions.
• Existing methods like Stochastic Gradient Langevin Dynamics (SGLD) can struggle with complex target distributions.

Purpose of the Study:

• To develop a scalable and efficient algorithm for Bayesian learning in big data settings.
• To address the limitations of existing methods in handling multi-modal and non-convex distributions.

Main Methods:

• Propose Contour Stochastic Gradient Langevin Dynamics (CSGLD), an adaptively weighted SGLD algorithm.
• CSGLD acts as a scalable dynamic importance sampler by adaptively flattening the target distribution.
• Theoretical analysis includes proving a stability condition and asymptotic convergence of the self-adapting parameter.

Main Results:

• CSGLD facilitates simulation for multi-modal distributions by adaptively flattening the target.
• Theoretical guarantees for stability and convergence of the adaptive parameter are established.
• Empirical results on CIFAR10 and CIFAR100 datasets demonstrate CSGLD's superiority over state-of-the-art algorithms.

Conclusions:

• CSGLD offers a robust and scalable approach for Bayesian learning in big data.
• The algorithm effectively handles complex distributions, leading to improved performance in deep neural network training.