Applicable or non-applicable: investigations of clinical heterogeneity in systematic reviews

Clinical heterogeneity can be defined as differences in participant characteristics, types or timing of outcome measurements and intervention characteristics. Clinical heterogeneity in systematic reviews has the possibility to significantly affect statistical heterogeneity leading to inaccurate conclusions and misled decision making. The aim of this study is to identify to what extent investigators are assessing clinical heterogeneity in both Cochrane and non-Cochrane systematic reviews.

The most recent 100 systematic reviews from the top five journals in medicine-JAMA, Archives of Internal Medicine, British Medical Journal, The Lancet, and PLOS Medicine-and the 100 most recently published and/or updated systematic reviews from Cochrane were collected. Various defined items of clinical heterogeneity were extracted from the included reviews. Investigators used chi-squared tests, logarithmic modeling and linear regressions to determine if the presence of such items served as a predictor for clinical heterogeneity when comparing Cochrane to non-Cochrane reviews. Extracted variables include number of studies, number of participants, presence of quantitative synthesis, exploration of clinical heterogeneity, heterogeneous characteristics explored, basis and methods used for investigating clinical heterogeneity, plotting/visual aids, author contact, inferences from clinical heterogeneity investigation, reporting assessment, and the presence of a priori or post-hoc analysis.

A total of 317 systematic reviews were considered, of which 199 were in the final analysis. A total of 81% of Cochrane reviews and 90% of non-Cochrane reviews explored characteristics that are considered aspects of clinical heterogeneity and also described the methods they planned to use to investigate the influence of those characteristics. Only 1% of non-Cochrane reviews and 8% of Cochrane reviews explored the clinical characteristics they initially chose as potential for clinical heterogeneity. Very few studies mentioned clinician training, compliance, brand, co-interventions, dose route, ethnicity, prognostic markers and psychosocial variables as covariates to investigate as potentially clinically heterogeneous. Addressing aspects of clinical heterogeneity was not different between Cochrane and non-Cochrane reviews.

The ability to quantify and compare the clinical differences of trials within a meta-analysis is crucial to determining its applicability and use in clinical practice. Despite Cochrane Collaboration emphasis on methodology, the proportion of reviews that assess clinical heterogeneity is less than those of non-Cochrane reviews. Our assessment reveals that there is room for improvement in assessing clinical heterogeneity in both Cochrane and non-Cochrane reviews.