Related Experiment Video

Updated: Jan 26, 2026

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
08:40

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity

Published on: June 12, 2019

7.9K

Risk of bias in nonrandomized studies of interventions showed low inter-rater reliability and challenges in its

Silvia Minozzi1, Michela Cinquini2, Silvia Gianola3

  • 1Department of Epidemiology, Lazio Regional Health Service, via Cristoforo Colombo 112, 00147 Rome, Italy; Department of Biomedical Sciences for Health, University of Milan, via Carlo Pascal 36, 20133 Milan, Italy.

Journal of Clinical Epidemiology
|April 15, 2019
PubMed
Abstract

Insights

The Risk of Bias In Nonrandomized Studies of Interventions (ROBINS-I) tool showed slight inter-rater reliability (IRR) and was found difficult to use. Intensive training is needed to improve IRR for this risk of bias assessment tool.

Area of Science:

  • Evidence-based medicine
  • Systematic reviews
  • Health research methodology

Background:

  • Assessing the risk of bias in nonrandomized studies of interventions is crucial for evidence synthesis.
  • The Risk of Bias In Nonrandomized Studies of Interventions (ROBINS-I) tool was developed to standardize this assessment.
  • Evaluating the usability and reliability of the ROBINS-I tool is essential for its effective implementation.

Purpose of the Study:

  • To evaluate the inter-rater reliability (IRR) of the ROBINS-I tool.
  • To assess the usability of the ROBINS-I tool among experienced raters.
  • To identify challenges and reasons for disagreement in applying the ROBINS-I tool.

Main Methods:

  • A cross-sectional study design was employed.
  • Five independent raters applied the ROBINS-I tool to 31 nonrandomized cohort studies from three systematic reviews.
  • Inter-rater reliability was measured using Fleiss' Kappa, and time taken to apply the tool was recorded.

Main Results:

  • Overall IRR for the ROBINS-I tool was slight (IRR 0.06).
  • IRR varied across domains, with the lowest for 'selection of reported results' (IRR 0.04) and highest for 'deviation from intended interventions' (IRR 0.18).
  • Raters reported difficulties due to poor primary study reporting, misunderstanding questions, and incomplete guidance, with an average application time of 27.8 minutes per study.

Conclusions:

  • The ROBINS-I tool was found to be difficult and demanding, even for experienced systematic reviewers.
  • Calibration exercises and intensive training are recommended prior to using the ROBINS-I tool to enhance reliability.
  • Improvements in reporting of primary studies may also enhance the usability and reliability of the ROBINS-I tool.
Keywords:
Inter-rater reliabilityNonrandomized studiesROBINS-IRisk of biasSystematic reviews

More Related Videos

How to Obtain Reliable Visual Event-related Potentials in Newborns07:39

How to Obtain Reliable Visual Event-related Potentials in Newborns

Published on: October 24, 2019

6.8K
Inter-Brain Synchrony in Open-Ended Collaborative Learning: An fNIRS-Hyperscanning Study04:44

Inter-Brain Synchrony in Open-Ended Collaborative Learning: An fNIRS-Hyperscanning Study

Published on: July 21, 2021

4.9K

Related Concept Videos

Bias in Epidemiological Studies01:29

Bias in Epidemiological Studies

Biases can arise at various stages of research, from study design and data collection to analysis and interpretation. Recognizing and addressing these biases is essential to ensure the validity and reliability of epidemiological findings.Broadly speaking, biases in epidemiology fall into three main categories: selection bias, information bias, and confounding. A more detailed description of possible biases is:  
1.3K
Confirmation Biases01:31

Confirmation Biases

The confirmation bias is the tendency to focus on information that confirms our existing beliefs and ignore information that is inconsistent with our expectations. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Have you ever fallen prey to the confirmation bias, either as the source or target of such bias?
8.1K
Hindsight Biases01:12

Hindsight Biases

Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Can you relate this to the phrase "Hindsight is 20/20" now? 
4.2K
Reliability and Validity01:29

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.8K
Bias01:22

Bias

Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
7.3K
Nursing Interventions I: Taxonomy of Nursing Interventions01:03

Nursing Interventions I: Taxonomy of Nursing Interventions

Nursing interventions are chosen as part of the planning process to achieve patient outcomes. Once nursing diagnoses are determined, the goals and outcomes are specified, then the nursing interventions are selected and individualized according to the patient's situation.
A nursing intervention is a treatment or action based on scientific concepts and knowledge from the nursing, behavioral, and physical sciences. Identifying and prioritizing nursing interventions based on the desired outcome...
3.7K