Posterior Covariance Information Criterion for Weighted Inference.

Area of Science:

• Statistics
• Machine Learning
• Information Theory

Background:

• Predictive evaluation is crucial for model selection and generalization.
• Existing information criteria like WAIC may not adequately address scenarios with differing estimation and evaluation likelihoods.
• Weighted likelihood inference, including covariate shift and counterfactual prediction, presents unique challenges for model evaluation.

Purpose of the Study:

• To develop a novel information criterion, the posterior covariance information criterion (PCIC), for predictive model evaluation.
• To generalize existing information criteria to effectively handle predictive scenarios with distinct estimation and evaluation likelihoods.
• To provide a computationally efficient criterion that can be calculated from a single Markov chain Monte Carlo (MCMC) run.

Main Methods:

• Development of the posterior covariance information criterion (PCIC).
• Utilizing a posterior covariance form for the criterion.
• Computation using a single Markov chain Monte Carlo (MCMC) run.
• Demonstration through numerical examples and theoretical analysis.

Main Results:

• PCIC effectively handles predictive scenarios where estimation and evaluation likelihoods differ.
• The criterion is computationally efficient, requiring only one MCMC run.
• PCIC is shown to be asymptotically unbiased to the quasi-Bayesian generalization error under mild conditions in weighted inference.
• The criterion is applicable to both regular and singular statistical models in weighted inference.

Conclusions:

• PCIC offers a robust and efficient method for predictive model evaluation, particularly in complex scenarios like weighted likelihood inference.
• The proposed criterion generalizes WAIC and addresses limitations in handling differing likelihoods for estimation and evaluation.
• PCIC provides a valuable tool for researchers and practitioners in statistics and machine learning for reliable model assessment.