Inter-rater reliability and concurrent validity of ROBINS-I: protocol for a cross-sectional study

Area of Science:

• Medical research methodology
• Evidence-based medicine

Background:

• The Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I) tool was developed by the Cochrane Bias Methods Group.
• Consistent application and interpretation of ROBINS-I are crucial for systematic reviews.
• Understanding the impact of training on ROBINS-I reliability is important.

Purpose of the Study:

• To establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the ROBINS-I tool.
• To identify barriers to using the ROBINS-I tool, such as evaluator burden and time commitment.
• To evaluate the effect of specialized training and guidance on ROBINS-I assessment reliability.

Main Methods:

• A cross-sectional study involving reviewers from four centers appraising non-randomized studies of interventions (NRSI).
• Two stages of assessment: without and with customized training and guidance.
• Inter-rater reliability and inter-consensus reliability calculated using Gwet's AC1 statistic; concurrent validity assessed against the Newcastle-Ottawa Scale (NOS) using Kendall's tau coefficient.

Main Results:

• The study aims to provide data on the reliability and validity of the ROBINS-I tool.
• It will quantify the impact of training on assessment consistency.
• Evaluator burden and time commitment for using ROBINS-I will be measured.

Conclusions:

• Findings will enhance the understanding and application of the ROBINS-I tool.
• This research will contribute to improving the quality of evidence from systematic reviews of NRSI.
• The study will inform best practices for using ROBINS-I in research synthesis.