Can AI improve triage quality? A preliminary assessment of ChatGPT performance in evaluating triage decisions

Ensuring accuracy and consistency in emergency department (ED) triage is vital to patient safety. Despite the presence of standardized protocols, variability in triage decisions remains a challenge. This study explores the potential of ChatGPT, a large language model (LLM), as a retrospective evaluator to assess the appropriateness of nurse-assigned triage levels according to the Tuscan Triage System (STT).

Fifty anonymized triage scenarios derived from an institutional quality-review and educational framework were included. Each scenario was independently reviewed and coded by two certified triage experts, with a third expert resolving any disagreement. ChatGPT (GPT-4o, OpenAI) was subsequently prompted to evaluate each scenario and determine whether the triage level originally assigned was appropriate. The model's assessments were compared with the expert-defined reference standard. Metrics included overall agreement, Cohen's Kappa, macro-averaged precision, recall, F1-score, and class-specific sensitivity and specificity.

Exact agreement between ChatGPT and expert assessments was found in 46% of cases. Discrepancies were more frequently under-triage (38%) than over-triage (16%). Overall agreement, measured by Cohen's Kappa, was 0.243. Performance was higher in high-complexity cases (κ = 0.313; F1 = 0.704), but decreased in moderate and low-complexity categories. Most misclassifications occurred between adjacent triage strata.

ChatGPT demonstrated moderate alignment with expert-assigned triage levels, particularly in critical cases. While not suitable for autonomous triage, the model shows potential as a retrospective quality assurance tool. Further refinement and clinical validation are required before integration into audit processes or decision support frameworks.