A Framework for Improving the Reliability of Black-box Variational Inference

Area of Science:

• Machine Learning
• Statistics
• Computational Statistics

Background:

• Black-box variational inference (BBVI) is a popular method for approximate Bayesian inference, offering speed and flexibility over traditional Markov chain Monte Carlo (MCMC) methods.
• However, existing stochastic optimization techniques for BBVI often lack reliability and require extensive manual tuning.
• This necessitates the development of more robust and automated approaches for practical application.

Purpose of the Study:

• To introduce Robust and Automated Black-box VI (RABVI), a novel framework designed to significantly improve the reliability of BBVI optimization.
• To provide a user-friendly system with minimal intuitive tuning parameters that automates complex optimization processes.
• To enable users to effectively balance computational cost with the desired accuracy of the variational approximation.

Main Methods:

• RABVI employs rigorously justified automation techniques for reliable optimization.
• It adaptively adjusts the learning rate upon detecting convergence of fixed-learning-rate iterates.
• The framework estimates symmetrized Kullback-Leibler (KL) divergence and uses a novel termination criterion to balance accuracy and computational cost.

Main Results:

• RABVI demonstrates improved robustness and accuracy in optimizing BBVI.
• The framework successfully detects inaccurate estimates of the optimal variational approximation.
• Simulation studies and real-world examples validate the effectiveness of RABVI.

Conclusions:

• RABVI offers a significant advancement in making BBVI more reliable and accessible for machine learning and statistical applications.
• The automated nature and adaptive learning rate adjustments reduce the need for expert knowledge and hand-tuning.
• The proposed termination criterion provides a practical way to manage the trade-off between accuracy and computational resources.