Statistically significant results from low-power analyses: A comedy of errors

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• Low statistical power in analyses can lead to overestimation of true effect sizes.
• The significance filter disproportionately selects high chance estimates, increasing bias as power diminishes.
• Estimation bias and type M error are key metrics for understanding this phenomenon.

Purpose of the Study:

• To quantify the estimation bias associated with low statistical power.
• To contrast this bias with type M error from a different perspective.

Main Methods:

• Simulations were used to quantify estimation bias in statistically significant results.
• Calculated were type M error, relative bias, and proportions of over/under-estimated results.

Main Results:

• At high power (≥80%), overestimation was moderate (relative bias <1.13) with accurate estimates common.
• At low power (<30%), overestimation was strong (relative bias >1.78), with few accurate estimates.
• Sign errors were prevalent only at very low power (<10%).

Conclusions:

• Statistically significant results from low-power analyses risk substantial overestimation (double effect).
• Magnitude errors, sign errors, and type 1 errors are common in low-power findings.
• Researchers should exercise caution when interpreting positive results from low-power studies.