Incorporating non-randomized evidence in cochrane intervention reviews: a scoping review

"Non-randomized studies of interventions" (NRSI) can provide valuable insights into the real-world performance of interventions, especially when randomized controlled trials (RCTs) are impractical, unethical, or lack generalizability. We investigated how Cochrane authors have incorporated evidence from NRSI in their reviews and whether this has changed over time.

We conducted a scoping review and identified Cochrane reviews, including NRSI which were published in the Cochrane Library in 2019 or 2023. We extracted data including how NRSI had been analyzed and assessed for risk of bias, and to what extent the authors had followed guidance in the Cochrane Handbook. This allowed us to identify the areas where review authors may need further guidance and support.

We identified 87 Cochrane reviews, 60 published in 2019 and 27 in 2023. In general, adherence to the guidance was low. Our key findings were that less than half of the reviews justified the inclusion of NRSI (36 reviews, 41%), less than a third stated prioritizing adjusted effect measures (25 reviews, 29%), and six analyzed RCTs and NRSI in the same meta-analysis, with no justification of this approach. Despite being the recommended tool for use in Cochrane reviews, only 25 reviews (29%) used the Risk Of Bias In Non-randomized Studies-of Interventions (ROBINS-I). We did find that adherence to the guidance improved between 2019 and 2023 but remained low.

Cochrane should consider how to increase the use of NRSI guidance, especially with the launch of Cochrane's Scientific Strategy, which may lead to an increase in the demand for reviews including NRSI evidence.