Leveraging Large Language Models for Automating Outpatients' Message Classifications of Electronic Medical Records.

Area of Science:

• Health Informatics
• Natural Language Processing
• Clinical Communication

Background:

• Healthcare generates vast unstructured text data, particularly from electronic medical record (EMR) portals.
• Efficient classification of outpatient messages is crucial for automating workflows and ensuring prompt clinical action.

Purpose of the Study:

• To evaluate the efficacy of large language models (LLMs) in classifying real-world outpatient messages.
• To compare general-purpose (GPT-4o) and domain-specific (BioBERT, ClinicalBERT) LLMs against a traditional baseline.
• To assess performance in both fine-tuned and few-shot learning scenarios.

Main Methods:

• Utilized de-identified outpatient messages from a central Illinois healthcare system.
• Compared GPT-4o, BioBERT, and ClinicalBERT models in fine-tuned and few-shot settings.
• Benchmarked LLM performance against a TF-IDF + Logistic Regression model.
• Conducted experiments within a HIPAA-compliant framework.

Main Results:

• Fine-tuned GPT-4o achieved 97.5% accuracy for urgency detection.
• Fine-tuned GPT-4o reached 97.8% accuracy for full message classification.
• GPT-4o models significantly outperformed BioBERT and ClinicalBERT across tested configurations.

Conclusions:

• Modern LLMs, particularly fine-tuned GPT-4o, are highly effective for outpatient communication triage.
• LLM application in this domain ensures both interpretability and compliance with privacy regulations.
• This technology holds significant potential for improving clinical response times and workflow efficiency.