About statistical significance, and the lack thereof.

Area of Science:

• Biostatistics
• Statistical Inference
• Clinical Trial Design

Background:

• Absence of statistical significance (p > 0.05) is often misinterpreted as evidence of no difference or treatment effect.
• Small sample sizes can lead to non-significant results, masking a true effect.

Purpose of the Study:

• To highlight the limitations of interpreting non-significant frequentist tests as absence of effect.
• To introduce and explain the utility of equivalence testing, specifically the two one-sided t-test, for establishing absence of meaningful effects.
• To demonstrate how equivalence tests can validate the biological relevance of statistically significant findings.

Main Methods:

• Explanation of the principles behind the two one-sided t-test (a form of equivalence test).
• Definition of the smallest effect size of interest (SESOI) as a prerequisite for equivalence testing.
• Illustrative examples of applying equivalence tests to interpret classical t-test results.

Main Results:

• Equivalence tests provide a statistically sound method to conclude the absence of a meaningful effect.
• The two one-sided t-test allows for correct interpretation when a classical t-test yields non-significant results.
• Equivalence tests can confirm if significant findings in large samples are also biologically meaningful.

Conclusions:

• Relying solely on p-values > 0.05 can lead to erroneous conclusions about the absence of effects.
• Equivalence testing, by pre-defining a minimal relevant difference, offers a robust approach to confirm the absence of meaningful effects.
• Integrating equivalence tests enhances the interpretation of statistical results, ensuring both statistical and practical significance.