Including nonrandomized evidence in living systematic reviews: lessons learned from the COVID-NMA initiative

Randomized controlled trials (RCTs) are more likely to be included in evidence syntheses of health interventions due to their methodological rigor. However, the integration of nonrandomized studies (NRSs) may be necessary, as was seen during the COVID-19 pandemic due to the emergence of variants of concern. We aimed to examine the body of evidence, randomized and nonrandomized, on COVID-19 vaccine effectiveness (VE) during the emergence of the Delta variant and to share lessons learned from including nonrandomized evidence alongside randomized evidence in the COVID-NMA living systematic review.

The COVID-NMA initiative is an international, living systematic review and meta-analysis that continually synthesized evidence on COVID-19 interventions. For this study, we identified all RCTs and comparative NRSs reporting on VE against the Delta variant from December 2020 (its initial detection) through November 2021 (date of last COVID-NMA NRS search). We conducted two parallel systematic reviews: one focusing on RCTs and the other on NRSs to compare available evidence on VE against the Delta variant. We also compared the publication timelines of the included studies with the global prevalence of the Delta variant, and documented the specific methodological challenges and solutions when including NRSs in living systematic reviews.

From December 2020 to November 2021, only one RCT reported vaccine efficacy against Delta in a subgroup of 6325 participants, while, during the same period, 52 NRSs including 68,010,961 participants reported VE against this variant. Nevertheless, including NRSs in our living systematic review posed several challenges. We faced difficulties in identifying eligible studies, encountered overlapping studies (ie, NRSs using the same database), and inconsistent definitions of Delta variant cases. Moreover, multiple analyses and metrics for the same outcome were reported without a pre-specified primary analysis in a registry or protocol. In addition, assessing the risk of bias required expertise, standardization, and training.

To remain responsive during public health emergencies, living systematic reviews should implement processes that enable the timely identification, evaluation, and integration of both randomized and nonrandomized evidence where appropriate.

When new health treatments are tested, the best way to see how well they work is through randomized controlled trials (RCTs). These are carefully designed studies that help reduce bias. However, during the COVID-19 pandemic, scientists also had to rely on other types of studies called nonrandomized studies (NRS) based on real-world data because the virus was changing quickly and required urgent action. Our living systematic review examined how effective COVID-19 vaccines were against the Delta variant, which spread widely from late 2020 to 2021. We wanted to understand what both RCTs and NRSs revealed about vaccine protection at that time. We also aimed to learn about the benefits and challenges of including different kinds of studies. From December 2020 to November 2021, we found that only one RCT reported results specifically for the Delta variant, including just over 6000 people. However, during the same period, 52 NRSs, involving over 68 million people, shared results about vaccine effectiveness against Delta in real-world settings. Including these NRSs were important for answering questions quickly, but it also created challenges. For instance, it was sometimes unclear how studies should be included, as many used the same data sources. Different studies defined "Delta cases" in various ways and often reported several kinds of results without stating which one was most significant. Evaluating the quality of these studies was complex and required special training. We developed rules to handle each of these challenges. In this study, we found that while RCTs remain the gold standard, NRSs provided crucial information during a fast-moving public health emergency. To help patients, doctors, and policymakers get timely answers in the future, living systematic reviews should be designed to include both types of evidence when appropriate, using clear methods to address challenges.