Exploration of Using an Open-Source Large Language Model for Analyzing Trial Information: A Case Study of Clinical...

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Biomedical Data Science

Background:

• Decentralized clinical trials (DCTs) are gaining interest, but analyzing their trends is challenging due to inconsistent data.
• Existing methods struggle with the heterogeneity and unstandardized terminology in trial registries.

Purpose of the Study:

• To explore the efficacy of the Llama 3 large language model for analyzing trends in decentralized clinical trials (DCTs).
• To assess the performance of different Llama 3 model sizes and fine-tuning on DCT data extraction and classification.

Main Methods:

• Utilized Llama 3 (8b, fine-tuned 8b, 70b) to analyze drug trial data from ClinicalTrials.gov (2018-2023).
• Employed prompt engineering for classifying DCTs and extracting decentralized elements.
• Evaluated model sensitivity and predictive value on a test dataset, then screened the full dataset.

Main Results:

• Fine-tuning improved model sensitivity from 0.0357 to 0.5385; further refinement increased positive predictive value to 0.9167.
• The 70b parameter Llama 3 model was essential for accurate extraction of decentralized elements.
• Identified 692 DCTs across the 6-year dataset, with a majority in Phase 2 (213) and Phase 4 (162).

Conclusions:

• Large language models show significant potential for analyzing unstructured clinical trial data, overcoming limitations of traditional methods.
• Careful management of potential biases is critical for reliable application of these AI models in clinical trial analysis.
• This approach facilitates a more efficient understanding of decentralized clinical trial trends and characteristics.