Harnessing Large-Language Models for Efficient Data Extraction in Systematic Reviews: The Role of Prompt Engineering.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Evidence-Based Medicine

Background:

• Systematic literature reviews (SLRs) are crucial for evidence-based medicine but face challenges with resource-intensive data extraction from randomized clinical trials (RCTs).
• Large language models (LLMs) offer potential for automating data extraction, but their efficacy across diverse medical conditions requires thorough investigation.

Purpose of the Study:

• To develop and evaluate prompt engineering strategies for GPT-4o to extract data from RCTs.
• To assess LLM performance in data extraction across three distinct disease areas: non-small cell lung cancer, endometrial cancer, and hypertrophic cardiomyopathy.

Main Methods:

• Iterative refinement of prompt engineering strategies for GPT-4o.
• Testing prompts on unseen RCT data and comparing LLM extraction with human extraction.
• Performance evaluation using F1 scores, precision, recall, and percentage accuracy.

Main Results:

• LLM demonstrated high effectiveness in extracting study and baseline characteristics, often matching human performance (test F1 scores > 0.85).
• Extraction of complex efficacy and adverse event data presented challenges, with lower test F1 scores (0.22–0.50).
• Promising, yet variable, transferability of prompts across disease areas necessitates disease-specific prompt refinement.

Conclusions:

• LLMs, enhanced by prompt engineering, can significantly augment the SLR process for data extraction.
• Human oversight remains indispensable for ensuring accuracy, particularly with complex and nuanced data.
• Ongoing validation of AI tools is essential for maintaining the quality and reliability of evidence synthesis as technology advances.