Counternull sets in randomized experiments

Area of Science:

• Statistics
• Biostatistics
• Clinical Trial Methodology

Background:

• Studies with non-statistically significant primary results are frequently misinterpreted as evidence of no effect.
• This misinterpretation can lead to flawed conclusions in scientific literature and clinical practice.

Purpose of the Study:

• To introduce and advocate for the reporting of "counternull" values alongside traditional null hypothesis testing.
• To demonstrate how counternull values can prevent the erroneous acceptance of null hypotheses when results are not statistically significant.

Main Methods:

• The study defines counternull values as non-null estimand values supported by the same evidence as the null value.
• Evidence is defined using randomization-based p-values from sharp null hypotheses in randomized experiments.
• A counternull set, rather than a single value, is proposed for reporting.

Main Results:

• A counternull set represents non-null effects that are statistically indistinguishable from the null effect based on the observed data.
• Reporting counternull sets can serve a pedagogical purpose, highlighting that non-rejection of the null does not equate to acceptance.
• Constructing counternull sets encourages deeper consideration of plausible effect sizes beyond the null.

Conclusions:

• Reporting counternull values offers a valuable supplement to standard p-values for interpreting non-significant findings.
• This approach enhances statistical rigor and promotes a more nuanced understanding of study results.
• The use of counternull values can improve the accuracy of conclusions drawn from clinical and scientific research.