A large language model-based clinical decision support system for syncope recognition in the emergency department: A...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Differentiating syncope from transient loss of consciousness (TLoC) presents diagnostic challenges in the emergency department (ED).
• Natural Language Processing (NLP) offers a method to analyze unstructured free text within electronic medical records (EMRs).

Purpose of the Study:

• To develop and evaluate large language models (LLMs) for accurate syncope recognition in the ED.
• To propose a framework for integrating these LLMs into clinical workflows for improved diagnostic support.

Main Methods:

• Development of two models using Italian and Multilingual Bidirectional Encoder Representations from Transformers (BERT) on consecutive EMRs.
• Training a "triage" model using only triage notes and an "anamnesis" model incorporating medical history data.
• Testing models on 15,098 (Italian BERT) and 15,222 (Multi BERT) EMRs, assessing performance using AUC and calibration plots.

Main Results:

• The "anamnesis" models achieved higher AUCs (0.98 for Italian BERT, 0.97 for Multi BERT) compared to "triage" models (0.95 for Italian BERT, 0.94 for Multi BERT).
• LLMs successfully identified syncope even when not explicitly stated in EMRs and recognized prodromal symptoms.
• Both models demonstrated high discriminative capability in identifying syncope patients from clinical notes.

Conclusions:

• LLM-based syncope recognition models, particularly the "anamnesis" model, exhibit high accuracy and discriminative power in the ED setting.
• These models can serve as valuable tools to aid physicians in differentiating syncope from other causes of TLoC.
• Integration of LLMs into clinical workflows holds promise for enhancing diagnostic efficiency and accuracy for syncope evaluation.