A test of the null hypothesis significance testing procedure correlation argument

Area of Science:

• Statistics
• Psychology
• Scientific Methodology

Background:

• The logic of null hypothesis significance testing (NHST) is debated, as it provides P(Data|H0) not P(H0|Data).
• NHST proponents argue P(Data|H0) correlates with P(H0|Data), justifying its use despite logical flaws.
• This presumed correlation lacks empirical investigation.

Purpose of the Study:

• To empirically test the correlation between P(Data|H0) and P(H0|Data).
• To determine if this correlation justifies the continued use of p-values in NHST.
• To examine how the correlation changes with stricter significance thresholds.

Main Methods:

• Simulated data under various null hypotheses.
• Calculated both P(Data|H0) and P(H0|Data) for each simulation.
• Quantified the correlation between these two probabilities across different significance levels.

Main Results:

• The correlation between P(Data|H0) and P(H0|Data) was found to be unimpressive.
• The observed correlation does not provide a compelling justification for computing p-values.
• The correlation weakened significantly as significance rules became more stringent (e.g., .01, .001).

Conclusions:

• The assumed correlation underpinning NHST's utility is empirically weak.
• P-values derived from NHST lack robust justification based on this correlation.
• Stricter significance criteria exacerbate the weakness of the NHST justification.