Perturbative partial moment matching and gradient-flow adaptive importance sampling transformations for Bayesian...

Area of Science:

• Bayesian inference
• Computational statistics
• Machine learning

Background:

• Leave-one-out (LOO) cross-validation is crucial for model evaluation in Bayesian statistics.
• Monte Carlo approximations for LOO predictions can be unstable, particularly with importance sampling (IS).
• Existing methods struggle with stabilizing importance weights in complex Bayesian models.

Purpose of the Study:

• To introduce a novel method for stabilizing Monte Carlo approximations of LOO cross-validated predictions.
• To improve the reliability and accuracy of Bayesian model evaluation.
• To address the instability of importance sampling weights in LOO predictions.

Main Methods:

• Developed gradient-flow-guided adaptive importance sampling (IS) transformations.
• Defined variational problems and derived nonlinear transformations using gradient information.
• Calculated Jacobian determinants with respect to the model Hessian for logistic regression and shallow neural networks.
• Proposed an approximation to avoid computing full Hessian matrices.

Main Results:

• The proposed transformations effectively stabilize importance weights for LOO predictions.
• Closed-form formulae for Jacobian determinants were derived for specific models.
• An approximation method was presented to bypass Hessian computation.
• The methodology demonstrated stability on a dataset known for unstable LOO IS weights.

Conclusions:

• Gradient-flow-guided adaptive IS transformations offer a robust solution for stabilizing LOO cross-validation in Bayesian models.
• The method enhances the reliability of Monte Carlo integration for predictive distributions.
• This approach provides a computationally feasible way to improve Bayesian model assessment.