Not normal: a simulation study comparing effect sizes for skewed psychological data

Area of Science:

• Psychological science
• Statistical methodology

Background:

• Effect sizes are crucial in psychological research for quantifying magnitude of effects.
• Current practice often prioritizes mean-based indices like Cohen's d, potentially overlooking distributional differences.

Purpose of the Study:

• To compare the performance of mean-based effect size indices (Cohen's d, CLES, ηₚ) with a non-parametric index (η) focusing on distributional differences.
• To evaluate index robustness under violations of normality and variance homogeneity.

Main Methods:

• Simulated data using skew-normal distributions to control mean differences, variance ratios, skewness, and sample size.
• Evaluated indices based on Relative Mean Bias, Normalized Root Mean Square Error, and 95% Coverage.

Main Results:

• Cohen's d demonstrated low bias, high precision, and accurate coverage across scenarios.
• CLES and ηₚ exhibited significant bias and low coverage, especially with skewness and heteroscedasticity.
• The non-parametric index η was unbiased under shape and variance differences but less reliable with population overlap.

Conclusions:

• Mean-based effect size indices like CLES and ηₚ are not interchangeable with Cohen's d and can be misleading.
• Cohen's d is a robust estimator for location differences, while η provides a comprehensive distributional perspective.
• Researchers should select effect sizes based on statistical properties and consider full distributional interpretation over mean-based conventions.