Risk-of-bias tools for causal inference studies using real-world data: a rapid scoping review

This review aimed to identify and characterize tools designed, adapted, or validated to assess risk of bias (RoB) in causal real-world evidence, with a view to informing their potential use in evidence-informed decision-making.

We conducted a rapid scoping review of records published from 2015 onwards. Eligible records included studies and documents describing tools explicitly developed for RWE or tools applicable to causal inference. Tool characteristics were extracted, and their items/domains were mapped across four stages of study generation and four core bias domains.

Eleven tools oriented toward causal questions were included, most of which were designed primarily for primary studies. Qualitative or categorical judgments were more common than numerical scoring. Coverage of the stages of study generation was uneven: study design was the most frequently addressed stage, whereas results presentation was the least represented. The data quality stage was addressed by ten tools, although its coverage varied across instruments. Across core bias domains, no tool was concentrated in a single domain; most distributed items across several domains, although the relative emphasis placed on selection and confounding bias varied across instruments. A substantial number of items/domains could not be classified under the predefined core bias domains.

Available RoB tools differ substantially in their structure, methodological emphasis, and the way they operationalize bias across stages of study generation and core bias domains. These differences should be considered when interpreting and comparing assessments conducted with different tools.

RWE uses information collected during routine healthcare to study whether healthcare interventions cause particular health outcomes. However, these studies may have problems in their design, data, or analysis that can lead to biased or misleading conclusions. Researchers use RoB tools to identify these potential problems and judge how trustworthy study results are. We reviewed 11 available tools that can be used to assess RoB in RWE with causality purpose. The tools differed considerably in what they assessed and how they evaluated potential bias. These differences should be considered when researchers and decision makers choose a tool or compare assessments made using different tools.