Assumption-checking rather than (just) testing: The importance of visualization and effect size in statistical...

Area of Science:

• Statistics
• Data Analysis
• Research Methodology

Background:

• Statistical methods rely on assumptions (e.g., normality) crucial for valid results.
• Violating these assumptions can lead to statistical errors and biased estimates.
• Current diagnostic practices for assumption checking are often problematic.

Purpose of the Study:

• To identify and illustrate the flaws in using null hypothesis significance tests for statistical assumption diagnostics.
• To provide practical recommendations for improving statistical diagnostic methods.

Main Methods:

• Analysis of a prevalent diagnostic approach using null hypothesis significance tests (e.g., Shapiro-Wilk test).
• Illustration of issues through simulations, focusing on statistical errors and misinterpretations.
• Synthesis of implications and development of recommendations for better diagnostics.

Main Results:

• Null hypothesis significance tests for assumption checking can yield false positives (large samples) and false negatives (small samples).
• This approach leads to false binarity, limited descriptiveness, and misinterpretation of p-values.
• Testing assumptions can fail if the test's own assumptions are unmet.

Conclusions:

• Rethink the reliance on significance testing for statistical assumption diagnostics.
• Emphasize visualization, effect sizes, and programmatic tools for more robust and replicable diagnostics.
• Distinguish between testing and checking assumptions, viewing violations as a spectrum.