Simplicity, complexity, and the standardized mean difference between two independent groups.

Area of Science:

• Psychometrics
• Statistical Inference
• Quantitative Research Methods

Background:

• Standardized effect sizes are common in research for comparing two independent groups.
• Common effect sizes include those proposed by Cohen and Glass.
• These measures face challenges, particularly 'the curse of the standardizer,' under unequal variances.

Purpose of the Study:

• To critically review existing effect size measures and confidence intervals.
• To propose a new heteroscedastic-consistent interval estimator for effect sizes.
• To evaluate the accuracy and robustness of the new estimator against existing methods.

Main Methods:

• Analysis of three common standardized effect sizes (Cohen's d, Glass's Δ).
• Development of a novel heteroscedastic-consistent confidence interval estimator.
• Empirical evaluation of estimator performance under non-normality and varying conditions.

Main Results:

• All three common effect sizes exhibit bias under assumption violations, especially non-normality.
• Existing confidence intervals show poor coverage rates.
• The proposed interval estimator demonstrates superior accuracy and robustness compared to traditional methods.
• Glass's Δ is found problematic even under recommended conditions.

Conclusions:

• Bias in effect size estimation is widespread, potentially leading to invalid inferences.
• The proposed confidence interval offers improved validity for effect size estimation.
• Further research is needed on bias-correction methods for non-normal data.
• Current practices using these effect sizes may have limited validity in many research scenarios.