Detecting and describing heterogeneity in meta-analysis

The investigation of heterogeneity is a crucial part of any meta-analysis. While it has been stated that the test for heterogeneity has low power, this has not been well quantified. Moreover the assumptions of normality implicit in the standard methods of meta-analysis are often not scrutinized in practice. Here we simulate how the power of the test for heterogeneity depends on the number of studies included, the total information (that is total weight or inverse variance) available and the distribution of weights among the different studies. We show that the power increases with the total information available rather than simply the number of studies, and that it is substantially lowered if, as is quite common in practice, one study comprises a large proportion of the total information. We also describe normal plots that are useful in assessing whether the data conform to a fixed effect or random effects model, together with appropriate tests, and give an application to the analysis of a multi-centre trial of blood pressure reduction. We conclude that the test of heterogeneity should not be the sole determinant of model choice in meta-analysis, and inspection of relevant normal plots, as well as clinical insight, may be more relevant to both the investigation and modelling of heterogeneity.