Large Language Models in Randomized Controlled Trials Design: Observational Study | JoVE Visualize

Area of Science:

Clinical Trial Design
Artificial Intelligence in Healthcare
Medical Research Methodology

Background:

Randomized controlled trials (RCTs) face significant challenges including limited generalizability, insufficient participant diversity, and high failure rates.
These limitations often stem from restrictive eligibility criteria and inefficient patient selection processes.
Large language models (LLMs) show potential in clinical applications, but their role in optimizing RCT design is largely unexplored.

Purpose of the Study:

To investigate the capability of LLMs, specifically GPT-4-Turbo-Preview, in assisting the design of RCTs.
To assess LLM's potential to improve RCT generalizability, recruitment diversity, and reduce failure rates.
To evaluate LLM-assisted RCT design while upholding clinical safety and ethical standards.

Main Methods:

An observational study analyzed 20 parallel-arm RCTs (10 completed, 10 registered) published after January 2024.
LLMs generated RCT designs based on provided criteria, including eligibility, recruitment, interventions, and outcomes.
Quantitative assessment by clinical experts and NLP metrics (BLEU, ROUGE-L, METEOR) evaluated LLM design accuracy against ClinicalTrials.gov data; qualitative assessments used Likert scales for safety, accuracy, bias, pragmatism, inclusivity, and diversity.

Main Results:

LLMs achieved 72% overall accuracy in replicating RCT designs, with high accuracy in recruitment (88%) and intervention (93%) design.
Lower accuracy was observed in designing eligibility criteria (55%) and outcomes measurement (53%).
Qualitative evaluations indicated strong clinical alignment, with LLM-generated designs ranking similarly to original designs in safety, accuracy, and objectivity, while enhancing diversity and pragmatism.

Conclusions:

LLMs demonstrate significant potential to enhance RCT design, particularly in recruitment and intervention strategies, improving generalizability and diversity.
Expert oversight and regulatory frameworks are crucial for ensuring patient safety and ethical compliance in LLM-assisted RCT design.
Further refinement of LLMs is needed to overcome limitations in eligibility criteria and outcomes measurement for broader clinical trial application.