Related Experiment Videos
Appraisal of Recent Intervention-Focused Umbrella Reviews Reveals Major Methodological Weaknesses: A
Pawel Posadzki1, Penny Whiting2, Lisa Hartling3
1Department of Clinical Rehabilitation, University of Physical Culture, Krakow, Poland.
Background:
Umbrella reviews (URs) are frequently used to provide high-level syntheses of evidence to inform clinical practice, policy, and guideline development. However, the methodological quality (MQ) and adherence to reporting standards of recent intervention-focused URs is understudied.
Objective:
To evaluate a random sample of recently published URs of interventions in health/medicine across several potential domains of MQ. Secondary objectives were to evaluate adherence to the Preferred Reporting Items for Overviews of Reviews (PRIOR) guideline and other recognised methodological guidance (e.g., Cochrane Handbook, JBI Manual).
Methods:
PubMed, Cochrane Library and Epistemonikos were searched in January 2026 for URs of interventions. We selected a random sample of 100 URs in health and/or medicine published between January 2024 and January 2026. Data were extracted pertaining to the following domains: identification and selection of systematic reviews (SRs), overlap of primary studies, bias inherited from included SRs, data extraction and evidence mapping, synthesis and interpretation bias, and reporting and transparency.
Results:
Of the 100 URs, 69% lacked a complete search strategy (median=4, IQR: 3-6 databases). Eligibility criteria were insufficiently reported in 96% of URs. Quantification of overlap was unclear in 77%, and only 11% considered overlap in synthesis. Only 21% of URs adhered to PRIOR. AMSTAR II was used in 77%, with AMSTAR I and JBI at 5% each. In 48%, inherited risk of bias from the SRs was high. Only 9% described how outdated or superseded SRs were managed and none managed discordant review conclusions. Outcome definitions were inconsistent in 71%, while only 9% avoided selective extraction of outcomes or analyses. Handling of multiple meta-analyses was unclear in 96%, and 71% selectively emphasised statistically significant results. Only 25% clearly described handling of heterogeneity and small-study effects, and 24% used evidence grading. 19% failed to register a protocol, and only 10% justified deviations from planned methods. Finally, 84% did not explicitly acknowledge umbrella-level limitations.
Conclusions:
We identified multiple methodological weaknesses in recent URs of interventions. Existing assessment tools appear only partially fit for purpose to assess the quality of the URs. Adherence to available guidance during the conduct and reporting of these recent URs is poor.
Related Concept Videos
01:29Bias in Epidemiological Studies
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
01:26Introduction to Epidemiology
01:25Strategies for Assessing and Addressing Confounding
Confounding can be addressed at both the design phase of a study and through analytical methods after data...