Scalable information extraction from free text electronic health records using large language models

Area of Science:

• Computational linguistics
• Health informatics
• Clinical research

Background:

• Electronic health records (EHRs) contain valuable free-text data on patient history.
• Extracting social determinants of health (SDoH) from unstructured EHR notes is challenging.
• Limited utility of SDoH data hinders research and healthcare improvements.

Purpose of the Study:

• To evaluate the accuracy of open-source large language models (LLMs) in extracting SDoH from clinical notes.
• To compare LLM performance against a traditional pattern-matching approach.
• To assess the feasibility of using LLMs without fine-tuning for SDoH data extraction.

Main Methods:

• Cross-sectional study using Mass General Brigham (MGB) EHR data.
• Manual labeling of SDoH aspects in a random sample of 200 patients.
• Evaluation of eight open-source LLMs against a baseline pattern-matching model using accuracy metrics and macro F1 scores.

Main Results:

• LLMs significantly outperformed the baseline pattern-matching model, especially for explicitly mentioned SDoH.
• The openchat_3.5 model demonstrated superior overall accuracy across all nine SDoH aspects.
• Prompt engineering in the LLM pipeline reduced hallucinations and improved data extraction accuracy.

Conclusions:

• Open-source LLMs provide an effective and scalable method for extracting SDoH from unstructured EHR data.
• LLMs surpass traditional pattern-matching techniques in SDoH data extraction.
• Further development, including domain-specific training, can enhance LLM utility in clinical research and predictive analytics for improved healthcare outcomes and reduced disparities.