Tuning-Free Coreset Markov Chain Monte Carlo via Hot DoG

Area of Science:

• Computational Statistics
• Machine Learning
• Bayesian Inference

Background:

• Bayesian coresets offer computational savings by approximating large datasets with smaller, weighted subsets.
• Current Coreset Markov chain Monte Carlo (MCMC) methods rely on stochastic gradient optimization, which is sensitive to learning rate hyperparameters.
• Suboptimal learning rates can degrade the quality of the coreset and its resulting posterior approximation.

Purpose of the Study:

• To develop a learning-rate-free optimization procedure for training Bayesian coreset weights.
• To enhance the robustness and reduce user tuning effort in Coreset MCMC algorithms.
• To improve the quality of posterior approximations obtained using Bayesian coresets.

Main Methods:

• Propose Hot-start Distance over Gradient (Hot DoG), a novel learning-rate-free stochastic gradient optimization algorithm.
• Provide a theoretical analysis of the convergence properties of coreset weights trained with Hot DoG.
• Empirically evaluate Hot DoG against existing learning-rate-free methods and adaptive optimizers like ADAM.

Main Results:

• Hot DoG successfully trains coreset weights without requiring manual learning rate selection.
• Theoretical analysis confirms the convergence of coreset weights generated by Hot DoG.
• Empirical results show Hot DoG yields superior posterior approximations compared to other learning-rate-free methods and is competitive with tuned ADAM.

Conclusions:

• Hot DoG offers a robust and user-friendly alternative for training Bayesian coresets within Coreset MCMC.
• The proposed method achieves high-quality posterior approximations without hyperparameter tuning.
• This advancement reduces computational cost and complexity in Bayesian inference with large datasets.