Bayesian model assessment and comparison using cross-validation predictive densities

Area of Science:

• Statistics
• Machine Learning
• Computational Science

Background:

• Assessing complex hierarchical Bayesian models is challenging.
• Estimating future predictive capability is crucial for model evaluation.
• Understanding uncertainty in model estimates is vital for reliable comparisons.

Purpose of the Study:

• To present practical methods for assessing, comparing, and selecting complex hierarchical Bayesian models.
• To introduce a framework for estimating and quantifying uncertainty in expected utility estimates.
• To enable robust model comparison through probabilistic assessments.

Main Methods:

• Utilizing cross-validation predictive densities for expected utility estimation.
• Employing Bayesian bootstrap for sampling from the distribution of expected utility estimates.
• Analyzing the properties of importance sampling and k-fold cross-validation techniques.

Main Results:

• Demonstrated the utility of expected utility distributions for model comparison.
• Showcased the effectiveness of the proposed cross-validation and Bayesian bootstrap approach.
• Successfully applied the methods to multilayer perceptron neural networks and Gaussian processes.

Conclusions:

• The proposed methods provide a robust framework for evaluating and selecting complex Bayesian models.
• Quantifying uncertainty in predictive performance is essential for reliable model assessment.
• The approach is applicable to various machine learning models and real-world problems.