Related Experiment Videos
Real-world evaluation of interconsensus agreement of risk of bias tools: A case study using risk of bias in
Samer Saadi1,2, Bashar Hasan1,2, Adel Kanaan1
1Evidence-Based Practice Center Mayo Clinic Rochester Minnesota USA.
Background:
Risk of bias (RoB) tools are critical in systematic reviews and affect subsequent decision-making. RoB tools should have adequate interrater reliability and interconsensus agreement. We present an approach of post hoc evaluation of RoB tools using duplicated studies that overlap systematic reviews.
Methods:
Using a back-citation approach, we identified systematic reviews that used the Risk Of Bias In Nonrandomized Studies-of Interventions (ROBINS-I) tool and retrieved all the included primary studies. We selected studies that were appraised by more than one systematic review and calculated observed agreement and unweighted kappa comparing the different systematic reviews' assessments.
Results:
We identified 903 systematic reviews that used the tool with 51,676 cited references, from which we eventually analyzed 171 duplicated studies assessed using ROBINS-I by different systematic reviewers. The observed agreement on ROBINS-I domains ranged from 54.9% (missing data domain) to 70.3% (deviations from intended interventions domain), and was 63.0% for overall RoB assessment of the study. Kappa coefficient ranged from 0.131 (measurement of outcome domain) to 0.396 (domains of confounding and deviations from intended interventions), and was 0.404 for overall RoB assessment of the study.
Conclusion:
A post hoc evaluation of RoB tools is feasible by focusing on duplicated studies that overlap systematic review. ROBINS-I assessments demonstrated considerable variation in interconsensus agreement among various systematic reviewes that assessed the same study and outcome, suggesting the need for more intensive upfront work to calibrate systematic reviewers on how to identify context-specific information and agree on how to judge it.
Related Concept Videos
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Bias in Epidemiological Studies
Bias
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
Randomized Experiments
Simple randomization
Simple...
Study Designs in Epidemiology
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and case-control studies.
Strategies for Assessing and Addressing Confounding
Confounding can be addressed at both the design phase of a study and through analytical methods after data...