AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs

Area of Science:

• Natural Language Processing (NLP)
• Biomedical Informatics
• Clinical Trial Analysis

Background:

• Systematic reviews are crucial for evidence-based medicine but are hindered by the increasing volume of clinical trial publications.
• Manual extraction of Population, Intervention, Comparator, and Outcome (PICO) elements is time-consuming and labor-intensive.
• Existing automated PICO extraction methods often require extensive manually annotated datasets, limiting their scalability.

Purpose of the Study:

• To develop and evaluate unsupervised and supervised Large Language Model (LLM) frameworks for automated PICO extraction from clinical trial reports.
• To leverage the pre-trained knowledge of LLMs for PICO term extraction in an unsupervised setting, reducing reliance on labeled data.
• To demonstrate the effectiveness of instruction tuning with Low Rank Adaptation (LORA) for high-performance PICO extraction in resource-constrained environments.

Main Methods:

• Utilized In-Context Learning (ICL) with a pre-trained LLM (AlpaCare) for unsupervised PICO term extraction.
• Employed instruction tuning with LORA on AlpaCare for supervised PICO term extraction, optimizing performance in low-resource settings.
• Evaluated both ICL and instruction-tuned frameworks on diverse PICO datasets (EBM-NLP, EBM-COMET, EBM-NLP_rev, EBM-NLP_h).

Main Results:

• The unsupervised ICL-based framework achieved comparable results across various EBM-NLP dataset versions.
• The instruction-tuned framework, utilizing LORA, demonstrated state-of-the-art performance on all tested EBM-NLP datasets.
• Both methods successfully extracted PICO-related terminologies from clinical trial documents, showcasing LLM efficacy.

Conclusions:

• LLM-based approaches, particularly instruction tuning with LORA, offer a powerful and efficient solution for automated PICO extraction.
• These methods significantly reduce the manual effort required for systematic reviews, accelerating evidence synthesis.
• The developed frameworks provide a scalable and effective tool for researchers analyzing clinical trial data.