Variational Hamiltonian Monte Carlo via Score Matching

Area of Science:

• Computational Statistics
• Bayesian Inference
• Machine Learning

Background:

• Computational Bayesian statistics traditionally splits into variational methods and Markov chain Monte Carlo (MCMC).
• Recent research combines variational Bayesian inference and MCMC for enhanced accuracy and efficiency.
• Scalable Bayesian inference methods benefit from integrating fast evaluation and flexible approximation.

Purpose of the Study:

• To incorporate variational approximation into Hamiltonian Monte Carlo (HMC) for reduced computational cost.
• To address the computational bottleneck in HMC for big data applications.
• To develop an efficient approximate Bayesian inference algorithm.

Main Methods:

• Exploited parameter space regularity to create a free-form approximation of the target distribution.
• Utilized an optimized additive model of proper random basis (akin to a single-hidden layer feedforward neural network) as a surrogate function.
• Employed the surrogate function for fast computation within the HMC sampling procedure.

Main Results:

• Developed a surrogate function that approximates the target distribution accurately and allows for fast computation.
• Achieved significant reduction in expensive computation during the HMC sampling process.
• Demonstrated the method's advantages on both synthetic and real-world datasets.

Conclusions:

• The proposed method offers an efficient approximate Bayesian inference algorithm by integrating variational approximation with HMC.
• This hybrid approach enhances computational efficiency and accuracy, making HMC more scalable for big data.
• The use of surrogate functions represents a promising direction for advancing Bayesian inference techniques.