Assessing the Reliability of Large Language Models for Evaluation of Risk of Bias in Randomized Clinical Trials

Systematic reviews depend on rigorous risk-of-bias (RoB) assessments to ensure credibility, yet manual evaluation using the Cochrane RoB 2 tool is resource-intensive. While large language models (LLMs) offer potential for automation, their alignment with human judgment remains underexplored. This study evaluates the reliability of ChatGPT-4o, ChatGPT-5, and Claude 3.5 Sonnet in assessing RoB in randomized controlled trials (RCTs), comparing their agreement with human reviewers and internal consistency.

We retrospectively analyzed 180 RCTs from systematic reviews published in the American Journal of Obstetrics and Gynecology (2021-2023) reporting complete human RoB 2 ratings. Each LLM processed full-text PDFs using a standardized prompt incorporating the complete RoB 2 algorithm. Model performance was evaluated against human benchmarks using Cohen's kappa and prevalence- and bias-adjusted kappa. Intramodel reliability was assessed across three independent runs to measure consistency.

ChatGPT-5 consistently outperformed other models, achieving the highest agreement in randomization (Domain 1; 76%), missing outcome data (Domain 3; 80%), and outcome measurement (Domain 4; 76%). It showed moderate concordance for deviations from intended interventions (69%). However, all models struggled with selective reporting (Domain 5), where agreement dropped to 47 to 51%. For overall RoB judgments, ChatGPT-5 demonstrated superior concordance (60-62%, κ = 0.36-0.40) compared with ChatGPT-4o (45%) and Claude 3.5 Sonnet (43%). ChatGPT-5 also exhibited substantial to near-perfect internal consistency.

Among the evaluated models, ChatGPT-5 most closely approximated human RoB 2 assessments and achieved superior internal consistency, suggesting it could serve as a practical first-pass tool to reduce reviewer burden. However, persistent limitations in detecting selective reporting-likely due to the inability to cross-reference external trial registries-highlight that expert human oversight remains essential for accurate evidence synthesis.

· GPT-5, GPT-4o, and Claude evaluated 180 RCTs.. · GPT-5 outperformed GPT-4o and Claude models.. · Models struggled with selective reporting bias..