Standardized Mean Differences: Not So Standard After All

Area of Science:

• Biostatistics
• Medical Research Methodology

Background:

• Standardized mean differences (SMDs), like Cohen's d and Hedges' g, are common in meta-analyses but are sensitive to within-study variability.
• This sensitivity can distort individual effect size estimates and impact overall meta-analytic findings.

Purpose of the Study:

• Introduce harmonized standardized mean differences (HSMDs) as a novel sensitivity analysis framework.
• Evaluate and address distortions in meta-analysis effect sizes caused by within-study sample variability.
• Enhance the comprehensiveness of meta-analytic synthesis.

Main Methods:

• Harmonize relative within-study variability across studies using the coefficient of variation (CV) to set empirical benchmarks.
• Recalculate SMDs under consistent variability assumptions.
• Apply the HSMD framework to meta-analytic data to assess the influence of study-specific standard deviations.

Main Results:

• Demonstrate the extent to which original effect sizes and pooled results are influenced by initial standardization variability.
• Quantify the impact of within-study variability on meta-analytic outcomes.
• Showcase the framework's ability to incorporate studies lacking reported variability metrics.

Conclusions:

• HSMDs provide a robust method for evaluating sensitivity to within-study variability in meta-analyses.
• This novel framework improves the reliability of effect size estimation and meta-analytic conclusions.
• The HSMD approach enhances meta-analytic synthesis by accommodating diverse study data.