Zero-Shot Extraction of Seizure Outcomes from Clinical Notes Using Generative Pretrained Transformers

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Artificial Intelligence in Medicine

Background:

• Pre-trained transformer models extract information from clinical notes but require manual annotation.
• Large Generative Pre-trained Transformer (GPT) models may streamline this process.
• Analyzing unstructured electronic health record (EHR) text is crucial for understanding population health trends.

Purpose of the Study:

• To explore GPTs in zero- and few-shot learning scenarios for analyzing clinical health records.
• To optimize prompt-engineered Llama2 13B for extracting seizure freedom from epilepsy clinic notes.
• To compare GPT performance against zero-shot and fine-tuned Bio+ClinicalBERT (BERT) models.

Main Methods:

• Prompt-engineered Llama2 13B model for seizure freedom extraction.
• Evaluation using zero-shot and few-shot learning scenarios.
• Comparison with zero-shot and fine-tuned Bio+ClinicalBERT models across various prompting paradigms.

Main Results:

• Zero-shot GPTs achieved promising median accuracy rates: one-word (62%), elaboration (50%), formatted dates (62%), and dates in context (74%).
• GPT performance surpassed zero-shot BERT (25%) but was lower than fine-tuned BERT (84%).
• In sparse contexts, the best-performing GPT (76%) outperformed fine-tuned BERT (67%) in seizure freedom extraction.

Conclusions:

• GPTs show potential for extracting clinically relevant information from unstructured EHR text without clinical annotation.
• GPTs offer insights into seizure management, drug effects, risk factors, and healthcare disparities.
• Prompt engineering enhances GPT accuracy, providing a framework for leveraging EHR data with zero annotation.