Addressing common inferential mistakes when failing to reject the null-hypothesis

Area of Science:

• Medical Statistics
• Clinical Research Methodology
• Biostatistics

Background:

• Null-hypothesis significance testing (NHST) can lead to erroneous conclusions about the absence of associations.
• Traditional statistical tests are limited in conclusively demonstrating the absence of an effect due to non-zero estimates and variances.
• Post-hoc power calculations provide no additional information beyond p-values and can be misleading.

Purpose of the Study:

• To highlight the limitations of traditional hypothesis testing in medical research.
• To advocate for the use of estimation accuracy for evaluating clinical relevance.
• To propose alternative approaches for interpreting statistical results in the presence of multiplicity.

Main Methods:

• Critique of traditional statistical tests and post-hoc power calculations.
• Emphasis on estimation accuracy for assessing clinical acceptability of associations.
• Discussion of p-value distributions and proportion of significant results for evidence evaluation.
• Exploration of limitations in multiplicity correction procedures.

Main Results:

• NHST cannot definitively prove the absence of an association.
• Estimation accuracy provides direct insights into clinical safety and efficacy.
• Post-hoc power calculations are uninformative and potentially misleading.
• Multiplicity corrections often fail to distinguish true from false positives.

Conclusions:

• Medical research should prioritize estimation accuracy over traditional hypothesis testing.
• Replication and consistency of findings are crucial for robust evidence.
• Focusing on the magnitude and variability of effects is more informative than binary true/false classifications.