Heterogeneous heterogeneity by default: Testing categorical moderators in mixed-effects meta-analysis.

Area of Science:

• Meta-analysis and statistical modeling
• Psychometrics and quantitative psychology

Background:

• Categorical moderators are frequently used in mixed-effects meta-analysis to explain heterogeneity in effect sizes.
• A common assumption is constant between-study variance across moderator levels, which can have significant statistical consequences.

Approach:

• Propose defaulting to unequal between-study variances for categorical moderator analysis.
• Introduce the mixed-effects location-scale model (MELSM) to estimate group-specific between-study variances.
• Conduct two extensive simulation studies to compare MELSM with equal variance mixed-effects models (MEM).

Key Points:

• MELSM shows minimal loss in Type I error and statistical power compared to MEM when variances are equal or sample sizes are balanced.
• With imbalanced sample sizes and unequal variances, MEM can lead to inflated or conservative Type I error rates, while MELSM maintains better control.
• MELSM generally offers similar or higher statistical power than MEM when MEM's Type I error rates are not inflated.

Conclusions:

• Assuming unequal between-study variances is a preferred default strategy for testing categorical moderators.
• MELSM provides a robust approach for handling between-study variance heterogeneity in moderator analyses.
• Researchers should consider adopting MELSM to improve the accuracy and reliability of moderator effect tests in meta-analysis.