Related Experiment Video

Updated: Apr 11, 2026

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

Published on: May 15, 2020

Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and

Clare Robertson¹, Craig Ramsay¹, Tara Gurung¹

¹Health Services Research Unit, University of Aberdeen, Aberdeen, UK.

Research Synthesis Methods

|June 9, 2015

Abstract

Unlabelled:

We describe our experience of using a modified version of the Cochrane risk of bias (RoB) tool for randomised and non-randomised comparative studies.

Objectives:

To assess time to complete RoB assessment. To assess inter-rater agreement. To explore the association between RoB and treatment effect size

Methods:

Cochrane risk of bias assessment was performed on a sample of full text primary reports included in a systematic review comparing operative techniques for radical prostatectomy. Inter-rater agreement was assessed using the kappa statistic.

Results:

Twenty-four studies were judged as high overall RoB, 13 were judged as low RoB and 11 were unclear. The weighted Kappa value was 0.35 indicating fair agreement. The median (range) time taken to rate each study was 30 min (10-49). The effect estimate for all studies was 0.61 (95% credible interval (CrI) 0.46-0.83) and 0.73 (95% CrI 0.29-1.75) for low risk studies.

Conclusions:

Although the process was time consuming, using a modified version of the RoB tool proved useful for demonstrating conservative effect estimates. That we only achieved a fair agreement between reviewers demonstrates the urgent need for further validation to improve inter-rater agreement. We suggest additional RoB levels could improve inter-rater reliability.

Keywords:

Cochrane Collaboration non-randomised risk of bias systematic review

More Related Videos

Inverse Probability of Treatment Weighting Propensity Score using the Military Health System Data Repository and National Death Index

Inverse Probability of Treatment Weighting Propensity Score using the Military Health System Data Repository and National Death Index

Published on: January 8, 2020

Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq

Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq

Published on: March 19, 2021

Related Concept Videos

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches

Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...

Study Designs in Epidemiology

Study Designs in Epidemiology

Epidemiological study designs are fundamental tools for investigating the distribution, determinants, and control of health conditions in populations. They help researchers understand the relationships between exposures and outcomes, and they broadly fall into two categories: "observational" and "experimental" studies.
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and...

Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs

Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs

Bioequivalence experimental study designs play a pivotal role in testing the effectiveness of various treatments. Key among these are the repeated measures, cross-over, carry-over, and Latin square designs. In the repeated measures design, each subject receives all treatments, allowing for temporal comparisons. This type of design is useful in reducing variability but requires careful planning to avoid bias.The cross-over design, an economical method, involves sequential administration of...

Hazard Ratio

Hazard Ratio

The hazard ratio (HR) is a widely used measure in clinical trials to compare the risk of events, such as death or disease recurrence, between two groups over time. It reflects the ratio of hazard rates—the instantaneous risk of the event occurring—between a treatment group and a control group. This measure provides valuable insights into the relative effectiveness of a treatment by assessing how the risk of an event differs between the two groups.
For example, in a clinical trial...

Bias in Epidemiological Studies

Bias in Epidemiological Studies

Biases can arise at various stages of research, from study design and data collection to analysis and interpretation. Recognizing and addressing these biases is essential to ensure the validity and reliability of epidemiological findings.Broadly speaking, biases in epidemiology fall into three main categories: selection bias, information bias, and confounding. A more detailed description of possible biases is:

Bioequivalence Experimental Study Designs: Completely Randomized and Randomized Block Designs

Bioequivalence Experimental Study Designs: Completely Randomized and Randomized Block Designs

Bioequivalence experimental study designs are crucial methodologies used in evaluating and comparing the bioavailability of different drug products. These designs are categorized into various types: completely randomized, randomized block, repeated measures, cross and carry-over, and Latin square designs.Completely randomized designs involve randomly allocating treatments to all subjects participating in the experiment. This allocation is achieved by assigning unique random numbers to subjects...

Articles linked to this work by shared authors, journal, and citation graph.

The 24-mo Outcomes from the Noninferiority Randomised Controlled Trial of Surgery for Men with Urodynamic Stress Incontinence After Prostate Surgery (MASTER).

European urology focus·2026

Concordance between automated/semi-automated measurement and manual assessment of mammographic breast density in individuals undergoing breast cancer screening: A systematic review.

Journal of medical screening·2026

Experiences of women from ethnic minorities and underserved, marginalised and disadvantaged groups in communicating with health professionals during antenatal care: An overview of qualitative systematic reviews.

NIHR open research·2026

Supplemental imaging modalities for breast cancer screening in women with dense breasts: A systematic review with economic considerations.

Breast (Edinburgh, Scotland)·2025

What are the influences on women's preferences and decision-making when planning mode of birth in high-income countries? A qualitative systematic review.

AJOG global reports·2025

A systematic review and network meta-analysis of interventions to preserve insulin-secreting beta cell function in people newly diagnosed with type 1 diabetes: results from randomised controlled trials of immunomodulatory therapies.

BMC medicine·2025

See all related articles