Concordance between an artificial intelligence self-triage programme and physical triage

We evaluated the concordance between the Dutch National Triage Standard (NTS) triage classification and the triage of an artificial intelligence (AI) self-triage programme.

This observational comparative study was performed in the emergency departments (EDs) of two OLVG hospital locations in Amsterdam in October and November 2023. Adult patients who entered the ED without ambulance transport and were triaged using NTS (usual care), were asked to complete the digital AI triage programme. Cohen's kappa (ĸ), per cent agreement and the distribution across urgency categories were used to evaluate the concordance between the triage methods.

Of the 264 patients approached, 203 consented to participation and were included in analysis and follow-up. Agreement between the triage methods was none to slight, ĸ=0.092 (95% CI -0.196 to 0.380). The AI triage programme overtriaged in 12.8% of cases and undertriaged in 5.4% compared with the NTS. However, the AI triage programme classified more patients with serious clinical sequelae into S1 (ie, emergency care with ambulance) and S2 (ie, emergency care) compared with NTS U1 and U2. The AI triage programme predicted the final diagnosis at ED discharge in 27.1% of the cases. There were significantly more clinical sequelae (eg, death, ward admission or follow-up consultation) for general practitioner referred patients compared with self-referrals.

The AI triage appeared more distinctive in urgency classification than classic triage when considering clinical sequelae for the patient. However, refinements and better validation of AI triage programmes are needed before these can be implemented in healthcare systems.