Diagnosing scientific replicability through probabilistic distinguishability.

Area of Science:

• Biological research
• Computational biology
• Genomics

Background:

• Replicability is crucial in biological research, yet computational tools to assess it are lacking.
• Existing methods struggle to quantify irreplicability and identify specific instances.
• This study addresses the need for robust computational approaches to evaluate research reproducibility.

Purpose of the Study:

• To develop an efficient and robust computational framework for quantifying and identifying irreplicable instances in biological research.
• To establish a criterion for distinguishing replicable yet heterogeneous effects from noise.
• To provide tools for detecting biases and uncovering biological heterogeneity.

Main Methods:

• Introduced a distinguishability criterion to define acceptable heterogeneity in replicable studies.
• Implemented a Bayesian model criticism approach with a Bayesian p-value for identifying irreplicable instances.
• Developed an R package, DiscRep, for practical application of the framework.

Main Results:

• Demonstrated the framework's efficacy in detecting batch effects in high-throughput experiments.
• Successfully identified instances of publication bias using the proposed methods.
• Applied the framework to GTEx eQTL data, revealing tissue-specific eQTLs and biological heterogeneity across tissues.

Conclusions:

• The developed framework provides an efficient and robust method for assessing research replicability.
• The approach can identify sources of irreproducibility, such as batch effects and publication bias.
• The application to eQTL data highlights the framework's utility in uncovering complex biological heterogeneity.