Averaging dependent effect sizes in meta-analysis: a cautionary note about procedures

Area of Science:

• Psychometrics
• Statistical Methods
• Meta-Analysis

Background:

• Primary studies often yield multiple effect sizes for a single construct.
• Averaging these dependent effect sizes is common in meta-analysis.
• Existing methods for averaging differ in how they handle effect size dependence.

Purpose of the Study:

• To compare three procedures for averaging dependent effect sizes: simple arithmetic mean, Hedges and Olkin (1985), and Rosenthal and Rubin (1986).
• To analyze the numerical and conceptual differences between these averaging procedures.
• To determine how these differences impact meta-analytic results.

Main Methods:

• The study simulated 54 conditions manipulating effect size magnitude and homogeneity, and correlation matrix properties.
• Three averaging procedures were applied to these simulated data: simple arithmetic mean, Hedges and Olkin, and Rosenthal and Rubin.
• The procedures were evaluated based on their numerical outputs and conceptual underpinnings.

Main Results:

• The Rosenthal and Rubin procedure yielded the highest effect size estimates.
• The simple arithmetic mean produced intermediate estimates.
• The Hedges and Olkin procedure resulted in the lowest effect size estimates.
• Differences in estimates were significant and dependent on the manipulated factors.

Conclusions:

• The choice of averaging procedure significantly impacts meta-analytic findings.
• The Rosenthal and Rubin procedure estimates effect size for a composite variable.
• The Hedges and Olkin procedure estimates effect size for a standard variable.
• Meta-analysts should select procedures based on study aims and data characteristics.