Killeen's probability of replication and predictive probabilities: how to compute, use, and interpret them

Area of Science:

• Psychological Science
• Bayesian Statistics
• Research Methodology

Background:

• The probability of replication (prep) is increasingly reported in psychological research.
• Existing guidance for prep calculation is limited, and the procedure lacks thorough scrutiny.
• Current implementations of prep assume known variance, limiting practical application.

Purpose of the Study:

• To identify and address practical issues in the computation and interpretation of prep.
• To provide conceptual and practical guidelines for the accurate use of prep.
• To extend prep to scenarios with unknown variance and explore other applications of predictive probabilities.

Main Methods:

• Analysis of common errors in prep calculation, particularly concerning p-values.
• Development of an extended method for prep computation with unknown variance.
• Exploration of fiducial Bayesian predictive probabilities for experimental design and monitoring.

Main Results:

• Prep is often miscalculated due to confusion between one-tailed and two-tailed p-values.
• Misinterpretation of prep as the probability of a same-sign and significant effect (psrep) is a significant risk.
• New methods are presented for handling unknown variance and for using predictive probabilities in experimental design.

Conclusions:

• Accurate computation and interpretation of prep are crucial for reliable scientific inference.
• The proposed guidelines and extensions enhance the practical utility of prep.
• Fiducial Bayesian predictive probabilities offer valuable tools for various stages of the research process, from design to analysis.