Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes

Area of Science:

• Meta-analysis
• Biostatistics
• Research Synthesis

Background:

• Selective reporting of statistically significant results compromises meta-analytic validity.
• Univariate methods for detecting bias assume independent effect sizes, which is often not the case.
• Studies frequently yield multiple, dependent effect size estimates, necessitating advanced analytical approaches.

Purpose of the Study:

• To evaluate univariate tests for selective reporting and small-study effects when dependent effect sizes are present.
• To assess the performance of Egger's regression test with robust variance estimation (RVE) and multilevel meta-analysis (MLMA) for handling dependent effect sizes.
• To compare these methods against traditional univariate tests and a three-parameter selection model (3PSM).

Main Methods:

• Monte Carlo simulations were employed to evaluate statistical test performance.
• Three univariate tests (trim and fill, Egger's regression, 3PSM likelihood ratio test) were assessed.
• Two Egger's regression variants (RVE, MLMA) were examined for handling dependent effect sizes.

Main Results:

• Ignoring effect size dependence inflated Type I error rates across all univariate tests.
• Egger's regression variants (RVE, MLMA) maintained Type I error rates with dependent effect sizes.
• The 3PSM likelihood ratio test did not fully control Type I error rates.
• Most methods showed limited power to detect selection bias unless bias was strong.

Conclusions:

• Standard univariate tests for selective reporting are unreliable when effect sizes are dependent.
• Egger's regression test, when adapted with RVE or MLMA, offers a more robust approach for dependent effect sizes.
• Careful consideration of effect size dependency is crucial for accurate meta-analysis and bias detection.