Expert agreement in prior elicitation and its effects on Bayesian inference

Area of Science:

• Psychology
• Statistics
• Bayesian Inference

Background:

• Prior distributions are crucial for Bayesian inference, quantifying pre-data uncertainty.
• Prior elicitation interviews experts to define these distributions, but can introduce subjectivity.
• Interpersonal variation in priors may impact hypothesis testing outcomes.

Purpose of the Study:

• To investigate the effect of interpersonal variation in elicited prior distributions on Bayes factor hypothesis tests.
• To quantify how different prior distributions influence Bayes factor results in psychological research.

Main Methods:

• Elicited prior distributions from six psychology experts.
• Re-analyzed 1710 psychology studies using elicited priors and default priors.
• Quantified variation in Bayes factors using measures of evidence concordance (direction, strength category, value).

Main Results:

• Bayes factors showed sensitivity to different prior distributions.
• Qualitative conclusions of hypothesis tests were generally unaffected by prior variations.
• Sensitivity analyses provided a template for Bayesian robustness checks.

Conclusions:

• While Bayes factors are sensitive to prior choices, elicited variations often do not change study conclusions.
• Researchers can use these findings to assess the impact of prior elicitation in their Bayesian analyses.
• The study offers a framework for Bayesian robustness analyses involving multiple expert priors.