P-value calibration in multiple hypotheses testing.

Area of Science:

• Statistics
• Statistical Inference

Background:

• P-values are standard measures of evidence against a null hypothesis.
• Current calibration methods, like Selke, Bayarri, and Berger, rely on a theoretical Uniform(0,1) null distribution, which is not always applicable.
• Matching frequentist and Bayesian inference requires accurate p-value calibration.

Purpose of the Study:

• To generalize existing p-value calibration methods.
• To propose a new calibration approach using an empirically estimated null distribution.
• To provide a stronger interpretative framework for p-values.

Main Methods:

• Generalizing the Selke, Bayarri, and Berger p-value calibration.
• Estimating a sampling null distribution directly from the data.
• Utilizing contexts like multiple testing to obtain empirical null distributions.

Main Results:

• The proposed method provides a generalized calibration for p-values.
• The new calibration uses an empirical null distribution instead of a theoretical one.
• The calibration remains a simple analytic formula, similar to the original method.

Conclusions:

• The generalized calibration offers a more robust interpretation of p-values.
• Empirical null distributions enhance the accuracy of p-value calibration.
• This approach strengthens the link between frequentist and Bayesian statistical interpretations.