Equivalence of the statistics for replicability and area under the ROC curve

Area of Science:

• Psychophysics
• Statistical modeling
• Experimental design

Background:

• The reproducibility of scientific findings is a critical concern.
• Existing statistical measures assess experimental outcomes but may lack direct interpretability regarding replicability.
• The receiver operating characteristic (ROC) curve is a well-established tool for evaluating binary classification performance.

Purpose of the Study:

• To demonstrate the equivalence between a novel replicability statistic, p(rep), and a traditional psychophysical measure.
• To highlight the theoretical underpinnings connecting these two statistical approaches.
• To provide a unified framework for understanding experimental reproducibility.

Main Methods:

• Mathematical derivation to establish the equivalence between p(rep) and the area under the ROC curve.
• Analysis of the theoretical relationship between statistical power, significance level, and effect sign reproducibility.
• Application of the area theorem from psychophysics to interpret the findings.

Main Results:

• The probability of replication statistic, p(rep), was shown to be mathematically identical to the area under the ROC curve.
• Both statistics quantify the probability that an experimental effect's sign will be correctly identified.
• The equivalence is rooted in the area theorem, linking psychophysical measurements to statistical inference.

Conclusions:

• The p(rep) statistic offers a direct measure of experimental effect sign replicability.
• The established area under the ROC curve serves as a proxy for effect sign reproducibility.
• These findings unify concepts in statistical inference and psychophysics, offering a clearer understanding of experimental reliability.