Extracting language information from clinical notes using large language models.

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Health Services Research

Background:

• Patient language proficiency is crucial for equitable care and research.
• Electronic health record (EHR) language data is often incomplete or inaccurate.
• Heterogeneous documentation practices hinder multi-institutional data use.

Purpose of the Study:

• Develop and evaluate a named entity recognition (NER) pipeline using large language models (LLMs).
• Accurately extract detailed patient language status from unstructured clinical notes.
• Enable scalable and generalizable language information extraction for research and practice.

Main Methods:

• Defined four language status categories: fluent use, partial ability, lack of understanding, and unrelated mentions.
• Annotated datasets from Yale New Haven Hospital (YNHH) and MIMIC-III.
• Evaluated proprietary (GPT-4o) and open-source (LLaMA3, BERT) LLMs in zero-shot and fine-tuning settings.
• Assessed cross-site generalizability.

Main Results:

• GPT-4o achieved high F1 scores (87% YNHH, 82% MIMIC) without fine-tuning.
• Fine-tuned open-source models (BERT, LLaMA3) showed comparable or superior performance.
• LLMs, especially LLaMA3, demonstrated stronger cross-institutional generalizability than traditional models.
• Unrelated language mentions were the most challenging category.

Conclusions:

• The NER framework accurately extracts nuanced language information from clinical narratives.
• High accuracy and generalizability support large-scale, language-focused research.
• Implications include improved patient-provider communication, interpreter services, and equitable healthcare.