A Scalable Bayesian Sampling Method Based on Stochastic Gradient Descent Isotropization.

Area of Science:

• Computational statistics
• Bayesian inference
• Machine learning

Background:

• Stochastic gradient MCMC (SGMCMC) is crucial for large-scale Bayesian modeling.
• Current SGMCMC methods face challenges with complex models like Deep Nets, particularly regarding noise covariance and non-vanishing learning rates.
• Satisfying assumptions on loss landscapes and SGMCMC noise behavior simultaneously is difficult.

Purpose of the Study:

• To propose a novel, practical SGMCMC method addressing current limitations.
• To ensure isotropic stochastic gradient (SG) noise.
• To utilize a fixed, analytically determined learning rate.

Main Methods:

• Developed a novel SGMCMC algorithm.
• Introduced a method to make the stochastic gradient (SG) noise isotropic.
• Determined a fixed learning rate analytically.
• Conducted extensive experimental validations.

Main Results:

• The proposed method successfully makes the SG noise isotropic.
• A fixed learning rate was analytically derived and implemented.
• Experimental results demonstrate the proposal's competitiveness with state-of-the-art SGMCMC algorithms.
• The method operates effectively with non-vanishing learning rates.

Conclusions:

• The novel SGMCMC approach offers a practical solution for large-scale Bayesian modeling.
• Isotropic SG noise and an analytically determined fixed learning rate enhance algorithm performance.
• This method is competitive with existing state-of-the-art SGMCMC techniques.