Evaluating LLM-based generative AI tools in emergency triage: A comparative study of ChatGPT Plus, Copilot Pro, and...

Area of Science:

• Emergency Medicine
• Artificial Intelligence
• Clinical Triage

Background:

• Rising global emergency department (ED) visits necessitate improved triage accuracy.
• Large Language Models (LLMs) show potential for enhancing triage processes.
• This study assesses AI tools against human performance in a high-volume urban ED.

Purpose of the Study:

• To evaluate the triage accuracy of ChatGPT and Copilot compared to trained physicians.
• To investigate the potential of AI in reducing human bias in ED triage.
• To address challenges posed by ED crowding.

Main Methods:

• A prospective observational study in an urban ED over one week.
• Adult patients were randomly enrolled; minors, trauma, and incomplete data were excluded.
• Clinical vignettes were created by an emergency medicine physician and compared against AI (ChatGPT, Copilot) and nurse triage decisions.

Main Results:

• Overall triage accuracy: Nurses 65.2%, ChatGPT 66.5%, Copilot 61.8% (no significant difference).
• AI tools significantly outperformed nurses in identifying high-acuity patients (87.8% ChatGPT, 85.7% Copilot vs. 32.7% nurses).
• AI demonstrated consistent accuracy across demographics, unlike nurses who were more prone to mistriage younger patients.

Conclusions:

• ChatGPT and Copilot excel at identifying high-acuity patients, surpassing traditional nurse triage.
• Real-time emergency department capacity data is vital for preventing overcrowding and ensuring quality care.
• AI integration in triage requires careful consideration of its impact on patient flow and care quality.