Artificial intelligence for detecting anaphylaxis in electronic medical records

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Allergy and Immunology

Background:

• Diagnosing anaphylaxis is challenging due to clinical complexity and underrecording in electronic medical records (EMRs).
• Inaccurate or delayed diagnosis increases the risk of recurrent, potentially fatal allergic reactions.
• Automated systems, like large language models (LLMs), offer a promising solution for improving diagnostic accuracy and patient safety.

Purpose of the Study:

• To evaluate the effectiveness of LLMs in autonomously identifying anaphylaxis diagnoses from EMR text.
• To enhance patient safety and optimize care delivery through AI-driven diagnostic support.
• To compare the diagnostic performance of different LLM configurations in detecting anaphylaxis.

Main Methods:

• LLMs (GPT 3.5, 4, and 4 Turbo) analyzed 969 medical texts in Brazilian Portuguese.
• Texts were annotated by expert physicians as either anaphylaxis-positive or negative.
• LLM diagnostic suggestions were compared against physician diagnoses using precision, sensitivity, specificity, and accuracy metrics.

Main Results:

• GPT 4 Turbo achieved 90.6% precision, 100% sensitivity, 99.5% specificity, and 99.5% accuracy using a primary prompt.
• Incorporating World Allergy Organization (WAO) criteria slightly improved older LLM models but did not enhance GPT 4 Turbo's precision.
• The LLM demonstrated a high Cohen kappa coefficient of 0.95, indicating strong agreement with expert diagnoses.

Conclusions:

• LLMs show significant potential for automating anaphylaxis diagnosis from EMR data.
• AI-powered tools can effectively support healthcare professionals in identifying critical allergic reactions.
• Automated diagnosis can lead to improved patient safety and more efficient healthcare delivery.