Extracting epilepsy-related information from unstructured clinic letters using large language models

Area of Science:

• Medical Informatics
• Natural Language Processing
• Epilepsy Research

Background:

• Electronic health records (EHRs) contain valuable unstructured data for healthcare research.
• Extracting specific epilepsy-related information from clinical notes using artificial intelligence remains an under-explored area.
• Large language models (LLMs) offer potential for advanced medical data extraction.

Purpose of the Study:

• To compare and apply open-source LLMs for extracting key epilepsy information from unstructured clinic letters.
• To optimize the use of EHRs as a data resource for epilepsy research.
• To evaluate the performance of different LLMs and extraction methodologies.

Main Methods:

• Utilized a dataset of 280 annotated clinic letters.
• Explored open-source LLMs (Llama and Mistral series) with direct, summarized, and contextualized extraction methods.
• Employed role-prompting and few-shot prompting techniques.
• Evaluated performance against a gold standard, fine-tuned models, and human annotations.

Main Results:

• Llama 2 13b achieved superior extraction performance (F1 scores: epilepsy type .80, seizure type .76, ASMs .90).
• Direct extraction consistently showed high performance across LLMs.
• LLMs outperformed MedCAT in extracting epilepsy-related information by .2 F1 score.

Conclusions:

• LLMs show significant potential for accurate medical information extraction in epilepsy research.
• Method selection is crucial for optimizing LLM performance in extracting data from unstructured clinical text.
• This approach enhances medical research and patient care through advanced NLP.