Calibrated Model Criticism Using Split Predictive Checks

Area of Science:

• Statistics
• Computational Statistics
• Bayesian Inference

Background:

• Assessing Bayesian model generalization to unobserved data is crucial.
• Existing methods like posterior predictive checks are often uncalibrated or lack generality.
• Model-specific derivations limit the practical application of many current checks.

Purpose of the Study:

• Introduce split predictive checks (SPCs) as a general-purpose class of predictive checks.
• Address the need for reliable and interpretable assessments of Bayesian model generalization.
• Develop a method that maintains usability while directly targeting predictive generalization.

Main Methods:

• SPCs involve splitting data into training and test subsets.
• Models are fitted to the training data and predictive discrepancies are evaluated on the test data.
• Asymptotic theory is developed for single SPCs and divided SPCs.

Main Results:

• Both single and divided SPCs yield asymptotically calibrated p-values, unlike posterior predictive checks.
• Single SPCs effectively identify substantial model misspecification.
• Divided SPCs demonstrate sensitivity to subtle departures from modeling assumptions.

Conclusions:

• SPCs offer a reliable, flexible, and computationally efficient approach to Bayesian model assessment.
• These checks can reveal issues with predictive generalization missed by other methods.
• SPCs provide interpretable p-values crucial for understanding model fit.