Using Monte Carlo experiments to select meta-analytic estimators

Area of Science:

• Statistical analysis
• Meta-analysis methodology
• Research methods

Background:

• Meta-analysis is crucial for synthesizing research findings across various disciplines.
• Selecting appropriate meta-analytic estimators is vital for accurate and reliable results.
• Existing research lacks comprehensive guidance on estimator selection based on empirical performance.

Purpose of the Study:

• To demonstrate the utility of Monte Carlo analysis for selecting meta-analytic estimators.
• To compare the performance of 11 common meta-analytic estimators across diverse research scenarios.
• To provide practical guidance for researchers in choosing the most suitable estimator for their specific study.

Main Methods:

• Conducted 1620 experiments simulating various research characteristics (sample size, effect size, heterogeneity, publication bias).
• Evaluated 11 common meta-analytic estimators based on bias, mean squared error (MSE), and coverage rates.
• Replicated simulation environments from four recent meta-analysis studies.

Main Results:

• Estimator performance varied significantly across different performance measures (bias, MSE, coverage).
• No single estimator consistently outperformed others; performance was context-dependent.
• Sample size and effect size heterogeneity were key factors influencing relative estimator performance.
• An estimator optimal for MSE might be suboptimal for coverage rates.

Conclusions:

• Monte Carlo analysis provides a robust framework for evaluating and selecting meta-analytic estimators.
• Observable research characteristics, such as sample size and effect heterogeneity, can guide estimator choice.
• This study offers practical insights for meta-analysts to improve the rigor and validity of their research synthesis.