Development of BERT-based large language models for emergency department triage using real-world conversations

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Accurate emergency department (ED) triage is crucial for effective resource allocation.
• Previous AI models for triage used summarized clinical data, limiting real-world applicability.
• Large language models (LLMs) offer potential for analyzing complex clinical conversations.

Purpose of the Study:

• To develop and evaluate LLMs trained on actual clinical conversations for patient urgency classification.
• To compare the performance of these LLMs against existing AI models and general-purpose chatbots.
• To assess the explainability of the developed LLMs in clinical decision-making.

Main Methods:

• Utilized a curated dataset of anonymized triage conversations from Korean tertiary hospitals.
• Developed two BERT-based LLMs: one tokenizing entire conversations, another using hierarchical structure with speaker-role embeddings.
• Classified patient urgency based on the Korean Triage and Acuity Scale (KTAS) into urgent (KTAS 3) and non-urgent (KTAS 4-5).
• Evaluated models using accuracy, precision, recall, and F1-score, comparing against ChatGPT GPT-4o and ClinicalBERT.
• Assessed model explainability using SHapley Additive exPlanations (SHAP).

Main Results:

• A hierarchical BERT-based LLM achieved 75.94% accuracy, significantly outperforming ChatGPT (56.68%) and ClinicalBERT (69.42%).
• The best model demonstrated a recall of 0.9610 for urgent cases, surpassing ChatGPT's recall of 0.5352.
• SHAP analysis confirmed the model's focus on clinically relevant cues aligned with KTAS criteria.

Conclusions:

• BERT-based LLMs trained on real-world ED conversations provide superior triage accuracy compared to general AI models.
• This approach enhances clinical decision support systems with interpretable and efficient AI.
• The findings highlight the potential of specialized LLMs for improving emergency care workflows.