Distinguishing between statistical significance and practical/clinical meaningfulness using statistical inference

Area of Science:

• Sport and Exercise Science
• Statistical Inference

Background:

• The Neyman-Pearson (N-P) significance-testing approach is dominant in sport and exercise science for hypothesis testing.
• N-P testing allows dichotomous decisions on null hypotheses with controlled error rates when applied correctly.
• However, N-P testing does not quantify hypothesis support or effect size, leading to potential misinterpretation.

Purpose of the Study:

• To critically evaluate the limitations of traditional Neyman-Pearson null hypothesis significance testing.
• To introduce and advocate for magnitude-based inference as a more informative statistical approach.
• To highlight how magnitude-based inference can better inform decisions regarding practical and clinical significance.

Main Methods:

• Discussion and critique of the Neyman-Pearson significance-testing framework.
• Exploration of Bayesian inference and its complexities.
• Presentation of magnitude-based inference as a pragmatic alternative, integrating Bayesian concepts.

Main Results:

• Neyman-Pearson testing primarily indicates the existence of non-zero effects, not their magnitude or practical value.
• Bayesian inference offers insights into hypothesis support but can be complex due to prior distributions.
• Magnitude-based inference effectively estimates true effect magnitudes and their clinical importance.

Conclusions:

• Traditional Neyman-Pearson null hypothesis significance testing has significant shortcomings in interpreting effect sizes and practical relevance.
• Magnitude-based inference provides a valuable, pragmatic approach for estimating effect magnitudes and their importance.
• Wider adoption of magnitude-based inference can advance scientific understanding by addressing the limitations of N-P testing.