Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias...

Area of Science:

• Medical research methodology
• Artificial intelligence in healthcare

Background:

• The Risk-of-Bias 2 (RoB2) tool presents implementation challenges, including low interrater reliability and high time demands.
• Large language models (LLMs) are being explored as potential aids for RoB2 implementation, but their effectiveness is not yet established.

Purpose of the Study:

• To evaluate the accuracy of LLMs in performing Risk-of-Bias 2 assessments.
• To explore the utility of LLMs as research assistants for bias evaluation in clinical trials.

Main Methods:

• Systematic search of Cochrane Library for RoB2-assessed reviews (October 2023).
• Selection of 46 randomized controlled trials (RCTs) from eligible reviews.
• Independent RoB2 assessment by 3 experienced reviewers and comparison with LLM judgments against Cochrane Reviews and reviewer consensus.

Main Results:

• LLMs achieved accuracy rates of 57.5%-70% compared to Cochrane Reviews and 65%-70% compared to human reviewers.
• LLM accuracy improved when judgments were derived from algorithmically processed signaling questions.
• LLMs completed assessments in an average of 1.9 minutes, significantly faster than human reviewers (31.5 minutes).

Conclusions:

• LLMs demonstrate commendable accuracy in RoB2 assessments when guided by structured prompts and reasoning.
• LLMs show strong potential as assistive tools for bias evaluation, complementing human expertise.
• Further research with larger studies and optimized prompting strategies is recommended to enhance LLM performance.