AI-assisted protocol information extraction for improved accuracy and efficiency in clinical trial workflows

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Clinical Trial Management

Background:

• Clinical trial protocols are becoming increasingly complex, leading to significant burdens on trial teams.
• Challenges in knowledge management and protocol amendments impact efficiency and compliance.
• Standardizing protocol content can enhance efficiency, documentation quality, and regulatory adherence.

Purpose of the Study:

• To evaluate an AI system employing generative Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) for automated clinical trial protocol information extraction.
• To compare the extraction accuracy of a clinical-trial-specific RAG process against standalone LLMs.
• To assess the operational impact of AI-assisted extraction on simulated Clinical Research Coordinator (CRC) workflows.

Main Methods:

• Developed and implemented a clinical-trial-specific RAG process using generative LLMs.
• Compared RAG extraction accuracy against publicly available, fine-tuned standalone LLMs using expert-supported annotations.
• Simulated CRC extraction workflows to evaluate AI-assistance impact on task completion time, cognitive demand, and user preference.

Main Results:

• The RAG process achieved significantly higher extraction accuracy (89.0%) compared to standalone LLMs (62.6%).
• AI-assisted tasks in simulated workflows were completed at least 40% faster.
• Users rated AI-assisted tasks as less cognitively demanding and strongly preferred them.

Conclusions:

• AI-assisted information extraction, particularly using RAG, demonstrates superior accuracy and efficiency for clinical trial protocols.
• AI tools can reduce the burden on trial teams, improving workflow and user experience.
• Integration of AI-assisted extraction into real-world clinical workflows warrants further validation for impact on study feasibility and timelines.