P-values - a chronic conundrum

Area of Science:

• Statistical methodology in medical research
• Hypothesis testing and significance evaluation

Background:

• The p-value is frequently misconstrued as the probability of a Type I error, leading to significant issues in research reproducibility and clinical decision-making.
• Common statistical education often conflates Fisher's significance testing with Neyman-Pearson's hypothesis testing, obscuring fundamental differences and contributing to widespread confusion.

Discussion:

• The p-value quantifies evidence against a null hypothesis, with smaller values indicating stronger evidence, but it is not the probability of a Type I error.
• A p-value of 0.05 does not imply a 5% chance of a Type I error; the actual probability of a treatment not working can be substantially higher, at least 28.9%.

Key Insights:

• Clarifying the distinction between statistical significance and hypothesis testing is crucial for accurate interpretation of research findings.
• The conventional p-value's misinterpretation has detrimental effects on treatment selection and empirical analysis.
• A practical alternative is needed to address the limitations of traditional p-value usage.

Outlook:

• Implementing calibrated p-values, representing the probability a treatment does not work, is essential for informed medical practice and research.
• Future research should focus on developing and disseminating intuitive, mathematically sound educational materials on statistical interpretation.
• Adopting calibrated p-values will enhance transparency and reliability in medical research and clinical decision-making.