Related Experiment Video
Updated: Sep 5, 2026
Setting Up a Stroke Team Algorithm and Conducting Simulation-based Training in the Emergency Department - A Practical Guide
Published on: January 15, 2017
Exposing and addressing the clinical safety gap in ESI-free emergency department triage: Imbalance-aware,
Jin Woo Jung1, Sami Souihi2, Matthieu Heidet3
1Department of Computer Science, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul, Republic of Korea.
Background:
AI-based emergency department (ED) triage must balance predictive performance, clinical safety, and interpretability. Prior models suffer from target leakage due to Emergency Severity Index (ESI) scores, or prioritize accuracy while failing to detect high-risk patients.
Objective:
To expose a pervasive clinical safety gap in ESI-free ED triage, accuracy-optimized models detecting fewer than 1% of high-risk patients despite AUROC near 0.80, and to show that imbalance-aware training is a broadly effective remedy across the evaluated model families.
Methods:
Using MIMIC-IV-ED (N = 407,735 adult visits; ESI excluded from inputs and labels), outcome-based labels were High Risk (expired, transferred, or 30-day mortality), Medium (admitted, survived), and Low (discharged). DA-V2, a dual-attention model trained with focal loss, mixup, and five-seed ensembling, was compared against seven baselines and an identically-trained MLP; a vitals-only variant was externally validated. Attention-SHAP concordance was assessed by Spearman correlation.
Results:
DA-V2 achieved AUROC 0.7949 (95% CI 0.7908-0.7991), outperforming all balanced tree baselines (p<0.001). An identically-trained MLP matched it (AUROC 0.7937, p=0.928; HR Recall 74.2% vs. 74.1%), showing that imbalance-aware training, not the architecture, drives detection. Focal loss with seed ensembling (mixup omitted, TabNet API) raised TabNet's HR Recall from under 1% to 44% (comparable to class-balanced TabNet, 45%) but did not reach the 74% attained only by the two dense models. Default models detected fewer than 1% of high-risk patients despite AUROC 0.796-0.800. Externally (NHAMCS-ED 2022, 12,598 adults; an approximate, proxy-label check), the vitals-only model retained HR Recall 0.759 at AUROC 0.695. Attention showed rank concordance with SHAP (Spearman ρ=0.708, p<10-12), evidence of rank agreement rather than causal faithfulness.
Conclusions:
A pervasive clinical safety gap, high AUROC with near-zero high-risk detection, is closed by imbalance-aware training and operating-point selection, not by any specific architecture. DA-V2 provides an interpretable, externally checked instantiation for trustworthy, ESI-free triage.
