Bias, coverage, and asymptotic behaviour of random effects meta-analysis: a clinically driven simulation study

The study of foundational features of meta-analysis is incomplete and continues to remain important. Using simulations we study bias and coverage and the asymptotic behaviour of the DerSimonian and Laird (D&L) meta-analysis with varying trial numbers and sizes, levels of risk, and extent of treatment effects.

With simulated data we model risk of untoward events in randomized controlled trials in meta-analyses. Treatment effect is expressed as relative risk reduction, with effect size estimated by the odds ratio which is then compared with the known population odds ratio. Performance is measured as bias, standardized bias and coverage, with thresholds for desirable results being prespecified.

Bias, standardized bias, and coverage varied substantially across meta-analyses of different trial size and number, risk mean and distribution, and relative risk reduction. Although improvements were observed with increasing trial size and number, there was widespread lack of satisfactory performance. Performance using normal risk distributions was worse compared with performance using constant or narrow uniform risk distributions. Asymptotic behaviour using very large trial numbers failed to show bias that appeared to approach zero for any distribution.

The D&L random effects meta-analysis method performed modestly at best. We were unable to demonstrate asymptotic normality. These results question the validity of the random effects method. The findings need replication and extension, which, if confirmed, would warn against generic use of the D&L method.