Standardized Mean Differences: Not So Standard After All

Meta-analyses often use standardized mean differences (SMDs), such as Cohen's d and Hedges' g, to compare treatment effects. However, these SMDs are highly sensitive to the within-study sample variability used for their standardization, potentially distorting individual effect size estimates and compromising overall meta-analytic conclusions. This study introduces harmonized standardized mean differences (HSMDs), a novel sensitivity analysis framework designed to systematically evaluate and address such distortions. The HSMD harmonizes relative within-study variability across studies by employing the coefficient of variation (CV) to establish empirical benchmarks (e.g., CV quartiles). SMDs are then recalculated under these consistent variability assumptions. Applying this framework to Meta-analytic data reveals the extent to which (original) effect sizes and pooled results are influenced by initial, study-specific standard deviations to standardize mean differences. Furthermore, the method facilitates the inclusion of studies lacking reported variability metrics into the sensitivity analysis, enhancing the comprehensiveness of the meta-analytic synthesis.