Multi-Evidence Clinical Reasoning With Retrieval-Augmented Generation for Emergency Triage: Retrospective Evaluation...

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems
• Emergency Medicine

Background:

• Emergency triage accuracy is vital but challenged by clinician experience, cognitive load, and case complexity.
• Mis-triage can lead to delayed care for high-risk patients and increased emergency department crowding.
• Large language models (LLMs) offer potential for triage decision support but face challenges with reliability and hallucinations.

Purpose of the Study:

• To evaluate a dual-source retrieval-augmented generation (RAG) system integrating guideline and case evidence for emergency triage.
• To compare the RAG system's performance against a baseline LLM.
• To assess the alignment of RAG system urgency assignments with expert consensus and clinical severity.

Main Methods:

• Development of a dual-source RAG system (MECR-RAG) using Hong Kong Accident and Emergency Triage Guidelines and 3000 triage cases.
• Retrospective evaluation of MECR-RAG and a prompt-only LLM on 236 triage encounters.
• Comparison of system predictions against expert consensus labels (senior triage nurses) using quadratic weighted kappa (QWK) and accuracy.

Main Results:

• MECR-RAG achieved superior performance (QWK 0.902, accuracy 0.802) compared to the baseline LLM (QWK 0.801, accuracy 0.542).
• MECR-RAG significantly reduced overtriage (12.7% vs 28.8%) while maintaining low undertriage (1.3% vs 1.7%).
• The system demonstrated improved sensitivity in detecting high-risk patients compared to initial nurse triage.

Conclusions:

• The dual-source RAG system (MECR-RAG) achieved expert-comparable agreement in emergency triage.
• MECR-RAG reduced overtriage and improved urgency assignment alignment compared to a baseline LLM.
• Further prospective studies are recommended to confirm effects on patient flow and outcomes.