Related Experiment Video
Updated: Feb 4, 2026

Author Spotlight: Evaluating the Adjuvant Efficacy and Safety of Angong Niuhuang Pill in Viral Encephalitis Treatment
Published on: April 19, 2024
Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized
Angelika Eisele-Metzger1,2, Judith-Lisa Lieberum3, Markus Toews1
1Institute for Evidence in Medicine, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany.
Abstract:
Systematic reviews are essential for evidence-based health care, but conducting them is time- and resource-consuming. To date, efforts have been made to accelerate and (semi-)automate various steps of systematic reviews through the use of artificial intelligence (AI) and the emergence of large language models (LLMs) promises further opportunities. One crucial but complex task within systematic review conduct is assessing the risk of bias (RoB) of included studies. Therefore, the aim of this study was to test the LLM Claude 2 for RoB assessment of 100 randomized controlled trials, published in English language from 2013 onwards, using the revised Cochrane risk of bias tool ('RoB 2'; involving judgements for five specific domains and an overall judgement). We assessed the agreement of RoB judgements by Claude with human judgements published in Cochrane reviews. The observed agreement between Claude and Cochrane authors ranged from 41% for the overall judgement to 71% for domain 4 ('outcome measurement'). Cohen's κ was lowest for domain 5 ('selective reporting'; 0.10 (95% confidence interval (CI): -0.10-0.31)) and highest for domain 3 ('missing data'; 0.31 (95% CI: 0.10-0.52)), indicating slight to fair agreement. Fair agreement was found for the overall judgement (Cohen's κ: 0.22 (95% CI: 0.06-0.38)). Sensitivity analyses using alternative prompting techniques or the more recent version Claude 3 did not result in substantial changes. Currently, Claude's RoB 2 judgements cannot replace human RoB assessment. However, the potential of LLMs to support RoB assessment should be further explored.
Related Concept Videos
meta-Directing Deactivators: –NO2, –CN, –CHO, –⁠CO2R, –COR, –CO2H
2° Amines to N-Nitrosamines: Reaction with NaNO2
Design Example: Analyzing Capacity Contours for Flood Risk Assessment
Confirmation Biases
SN2 Reaction: Kinetics
In a chemical reaction, a relationship exists between the concentration of reactants and the rate at which the reaction proceeds. The study to measure this relationship is known as the kinetics of a chemical reaction. Kinetic studies are used to deduce the rate law of a chemical reaction, which provides information about the species involved during the transition state of the rate-determining step. Thus, kinetic studies help to derive the mechanism of a...
SN2 Reaction: Mechanism
The presence of the more electronegative halogen in the substrate creates a polarized carbon-halide bond. The halide pulls the electron cloud generating an electrophilic center at the carbon atom. Thus, the carbon atom carries a partial positive charge while the halide has a...

