Calibrating Bayesian inference

Area of Science:

• Psychological statistics
• Bayesian inference
• Computational statistics

Background:

• Bayesian statistics is increasingly used in psychology for its uncertainty quantification.
• However, priors are often used computationally, not as genuine beliefs, which can be conceptually ungrounded.
• Standard Bayesian inference can be misleading when prior assumptions mismatch the true data-generating process.

Purpose of the Study:

• To propose an alternative justification for "pragmatic Bayes" through finite-sample performance evaluation.
• To introduce a method for calibrating Bayesian credible regions to achieve frequentist validity.
• To address the vulnerability of standard Bayesian inference to prior-data mismatch.

Main Methods:

• Evaluating finite-sample performance over repeated sampling of data and parameters.
• Developing a novel stochastic approximation algorithm for calibrating Bayesian credible regions.
• Conducting Monte Carlo experiments to compare calibrated and uncalibrated Bayesian inference.

Main Results:

• Uncalibrated Bayesian inference can be liberal (i.e., overly confident) under certain data-generating scenarios.
• The proposed calibration method consistently maintains frequentist validity, regardless of the underlying parameter-generating mechanism.
• The calibration procedure was illustrated using a real-data example in location-scale regression.

Conclusions:

• Finite-sample performance evaluation offers a pragmatic justification for Bayesian methods in psychology.
• Calibrating Bayesian credible regions to frequentist standards provides robust and valid inference.
• The novel stochastic approximation algorithm effectively solves the practical calibration problem.