A comparison of hypothesis tests for homogeneity in meta-analysis with focus on rare binary events

Analysis of rare binary events is an important problem for biomedical researchers. Due to the sparsity of events in such problems, meta-analysis that integrates information across multiple studies can be applied to increase the efficiency of statistical inference. Although it is critical to examine whether the effect sizes are homogeneous across all studies, a comprehensive review of homogeneity tests has been lacking, and in particular, no attention has been paid to infrequent dichotomous outcomes. We systematically review statistical methods for homogeneity testing. By conducting an extensive simulation analysis and two case studies, we examine the performance of 30 tests in meta-analysis of rare binary outcomes. When using log-odds ratio as the association measure, our simulation results suggest that there is no uniform winner. However, we recommend the test proposed by Kulinskaya and Dollinger (BMC Med Res Methodol, 2015, 15), which uses a gamma distribution to approximate the null distribution, for its generally good performance; for very rare events coupled with small within-study sample sizes, in addition to the Kulinskaya-Dollinger test, we further recommend the conditional score test based on the random-effects hypergeometric model proposed by Liang and Self (Biometrika, 1985, 72:353-358). One should be cautious about the use of the Wald tests, the Lipsitz tests (Biometrics, 1998, 54:148-160), and tests proposed by Bhaumik et al (J Am Stat Assoc, 2012, 107:555-567).