Incorporating variations in the quality of individual randomized trials into meta-analysis

Meta-analysis is a method of synthesizing evidence from multiple sources. It has been increasingly applied to combine results from randomized trials of therapeutic strategies. Unfortunately there is often variation in the quality of the trials that are included in meta-analyses, limiting the value of combining the results in an overview. This variation in quality can lead to both bias and reduction in precision of the estimate of the therapy's effectiveness. There are a number of methods for quantifying the quality of trials including the detailed Chalmers system and simple scales. The nature of the relationship between these quality scores and the true estimate of effectiveness is unknown at this time. We discuss four methods of incorporating quality into meta-analysis: threshold score as inclusion/exclusion criterion, use of quality score as a weight in statistical pooling, visual plot of effect size against quality score and sequential combination of trial results based on quality score. The last method permits an examination of the relation between quality and both bias and precision on the pooled estimates. We conclude that while it is possible to incorporate the effect of variation of quality of individual trials into overviews, this issue requires more study.