Methodological evaluation of bias in observational coronavirus disease 2019 studies on drug effectiveness

Observational studies may provide valuable evidence on real-world causal effects of drug effectiveness in patients with coronavirus disease 2019 (COVID-19). As patients are usually observed from hospital admission to discharge and drug initiation starts during hospitalization, advanced statistical methods are needed to account for time-dependent drug exposure, confounding and competing events. Our objective is to evaluate the observational studies on the three common methodological pitfalls in time-to-event analyses: immortal time bias, confounding bias and competing risk bias.

We performed a systematic literature search on 23 October 2020, in the PubMed database to identify observational cohort studies that evaluated drug effectiveness in hospitalized patients with COVID-19. We included articles published in four journals: British Medical Journal, New England Journal of Medicine, Journal of the American Medical Association and The Lancet as well as their sub-journals.

Overall, out of 255 articles screened, 11 observational cohort studies on treatment effectiveness with drug exposure-outcome associations were evaluated. All studies were susceptible to one or more types of bias in the primary study analysis. Eight studies had a time-dependent treatment. However, the hazard ratios were not adjusted for immortal time in the primary analysis. Even though confounders presented at baseline have been addressed in nine studies, time-varying confounding caused by time-varying treatment exposure and clinical variables was less recognized. Only one out of 11 studies addressed competing event bias by extending follow-up beyond patient discharge.

In the observational cohort studies on drug effectiveness for treatment of COVID-19 published in four high-impact journals, the methodological biases were concerningly common. Appropriate statistical tools are essential to avoid misleading conclusions and to obtain a better understanding of potential treatment effects.