Bayesian prediction intervals for assessing P-value variability in prospective replication studies.

Area of Science:

• Statistical methodology
• Genomic data analysis
• Psychiatric research

Background:

• The increasing reliance on statistical analysis in scientific research highlights concerns about the misuse and limitations of statistical methods.
• P-values, widely used in hypothesis and significance testing, face scrutiny for their unsuitability as measures of hypothesis credibility and their inherent randomness.
• Current frequentist prediction intervals for P-value variability rely on unrealistic assumptions, leading to biased results and lack of interpretable probabilistic bounds for replication studies.

Purpose of the Study:

• To address the limitations of frequentist P-intervals by proposing a novel Bayesian approach for predicting P-value variability.
• To develop interpretable probabilistic bounds for P-values in prospective replication studies.
• To create a method resistant to selection bias in P-value prediction.

Main Methods:

• Development of Bayesian intervals for predicting P-value variability.
• Incorporation of prior knowledge about effect size distribution into interval construction.
• Application of the proposed Bayesian intervals to P-values from the Psychiatric Genomics Consortium's study on five psychiatric disorders.

Main Results:

• The proposed Bayesian intervals provide directly interpretable probabilistic bounds for replication P-values.
• These Bayesian intervals are resistant to selection bias, unlike traditional frequentist methods.
• The approach was successfully demonstrated using real-world genomic data from psychiatric research.

Conclusions:

• Bayesian prediction intervals offer a more reliable and interpretable method for assessing P-value variability and predicting replication outcomes.
• This approach enhances the trustworthiness of statistical conclusions drawn from scientific studies, particularly in large-scale genomic research.
• The proposed method provides a valuable tool for researchers seeking to understand and account for the inherent randomness in P-values.