Related Experiment Videos
Consensus operational criteria for including systematic reviews in umbrella reviews: an international Delphi study
Beth Nichol1, Jemma McCready2, Rebecca Gibson2
1School of Communities and Education, Northumbria University, Newcastle upon Tyne, UK and Policy Research Unit Behavioural and Social Sciences, Newcastle University, Newcastle upon Tyne, UK.
Objective:
This international, multidisciplinary study aimed to develop consensus-based operational criteria for umbrella reviewers to apply when including systematic reviews.
Introduction:
As the volume of academic publications continues to increase, umbrella reviews (systematic reviews of systematic reviews) are increasingly used to synthesize large bodies of evidence and provide high-level overviews. However, in the absence of formal guidance, the inclusion of articles based on study type within umbrella reviews is inconsistent, leading to heterogenous criteria for the inclusion of systematic reviews.
Methods:
A classic Delphi method was conducted anonymously online with an expert panel in systematic and umbrella reviews. Experts had published at least 5 umbrella or systematic reviews as first author or at least 10 at any authorship position. In round 1, 68 experts rated agreement with 188 elements of a systematic review identified in a previous study. Elements achieving ≥80% agreement progressed to round 2, where experts rated their certainty regarding inclusion in the operational criteria. Statements were collated into draft criteria and refined across rounds 3 to 5. Free-text comments were incorporated throughout. The consensus threshold was ≥80% initially and reduced to ≥70% after round 4.
Results:
Thirty-six experts completed all 5 rounds. In round 1, 25 elements met the ≥80% agreement threshold and were condensed for round 2. In round 2, 19 statements reached ≥80% certainty for inclusion and were combined into operational criteria in round 3, with consensus achieved in round 4 (70.3%). Agreement decreased in round 5 (66.7%), indicating a ceiling effect.
Conclusions:
The final consensus comprises criteria distinguishing systematic reviews as explicit and reproducible. Seven criteria are divisible into 12 individual checklist items, spanning the conduct and reporting of an article's rationale; overall methods; inclusion criteria; search strategy; and approach to screening, extraction, and synthesis. While some characteristics were contested (including operationalizing independent screening and a comprehensive search), the resulting operational criteria provide a practical tool to support transparent and consistent inclusion decisions in umbrella reviews.
Review Registration:
OSF https://osf.io/pxtbd.