From RAGs to riches: Utilizing large language models to write documents for clinical trials

Area of Science:

• Artificial Intelligence in Clinical Research
• Natural Language Processing Applications

Background:

• Clinical trials necessitate extensive documentation, including protocols, consent forms, and study reports.
• Large language models (LLMs) show promise for accelerating the drafting of these essential documents.
• Concerns exist regarding the quality and reliability of LLM-generated content for clinical trial documentation.

Purpose of the Study:

• To evaluate the efficacy of off-the-shelf large language models in generating clinical trial protocol sections.
• To assess the impact of retrieval-augmented generation (RAG) on improving LLM performance for clinical trial writing.
• To determine the practical usability of enhanced LLMs in addressing drug development bottlenecks.

Main Methods:

• Generated clinical trial protocol sections using an off-the-shelf LLM across various diseases and trial phases.
• Assessed generated sections on clinical thinking, logic, transparency, references, terminology, and content relevance.
• Implemented retrieval-augmented generation (RAG) by integrating accurate, up-to-date information, including regulatory guidance and ClinicalTrials.gov data.
• Regenerated protocol sections using the RAG-enhanced LLM and reassessed them using the same criteria.

Main Results:

• Off-the-shelf LLMs scored over 80% for content relevance and medical terminology but below 40% for clinical thinking, logic, and references.
• Retrieval-augmented generation (RAG) significantly improved performance, increasing scores for clinical thinking, logic, and references to approximately 80%.
• RAG substantially enhanced the practical usability of LLMs for generating clinical trial-related documents.

Conclusions:

• Hybrid LLM architectures, such as RAG, demonstrate strong potential for clinical trial writing.
• RAG-enhanced LLMs can effectively generate various clinical trial documents, addressing key bottlenecks in drug development.
• This approach offers a potentially transformative solution for improving the efficiency of clinical research documentation.