Randomized -values for multiple testing and their application in replicability analysis

Area of Science:

• Statistics
• Genomics
• Reproducibility Research

Background:

• Simultaneous testing of multiple endpoints presents a multiple testing problem with composite null hypotheses.
• Traditional p-values computed under least favorable parameter configurations (LFCs) are overly conservative for composite null hypotheses.
• This conservatism complicates the estimation of the proportion of true null hypotheses, a key aspect of replicability analysis.

Purpose of the Study:

• To investigate the application of randomized p-values for testing replicability hypotheses simultaneously across multiple endpoints.
• To introduce a general class of statistical models enabling easy calculation of valid randomized p-values.
• To compare the accuracy of randomized p-values against LFC-based approaches for estimating the proportion of true null hypotheses.

Main Methods:

• Utilized randomized p-values as an alternative to traditional p-values in the context of multiple testing.
• Developed a general class of statistical models amenable to randomized p-value computation.
• Employed computer simulations to evaluate the performance of the proposed methodology.
• Applied the methodology to a real-world genomics dataset.

Main Results:

• Randomized p-values provide a more accurate estimation of the proportion of true null hypotheses compared to LFC-based methods.
• The proposed statistical models facilitate straightforward calculation of valid randomized p-values.
• Simulations confirmed the superior performance of randomized p-values in replicability analysis.

Conclusions:

• Randomized p-values offer a valuable solution to the over-conservatism of traditional p-values in multiple testing scenarios with composite null hypotheses.
• The proposed methodology enhances the accuracy of estimating the proportion of true null hypotheses in replicability studies.
• The application to genomics data demonstrates the practical utility of randomized p-values in real-world scientific research.