Related Experiment Video
Updated: Jul 26, 2026
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
Interrater Reliability, Accuracy, and Triage Time Pre- and Post-implementation of a Real-Time Electronic Triage
Shelley L McLeod1, Joy McCarron2, Tamer Ahmed2
1Schwartz/Reisman Emergency Medicine Institute, Sinai Health System, Toronto, Ontario, Canada; Department of Family and Community Medicine, Division of Emergency Medicine, University of Toronto, Toronto, Ontario, Canada; Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada.
Study Objective:
The electronic Canadian Triage and Acuity Scale (eCTAS) is a real-time electronic triage decision-support tool designed to improve patient safety and quality of care by standardizing the application of the Canadian Triage and Acuity Scale (CTAS). The objective of this study is to determine interrater agreement of triage scores pre- and post-implementation of eCTAS.
Methods:
This was a prospective, observational study conducted in 7 emergency departments (EDs), selected to represent a mix of triage documentation practices, hospital types, and patient volumes. A provincial CTAS auditor observed triage nurses in the ED pre- and post-implementation of eCTAS and assigned an independent CTAS score in real time. Research assistants independently recorded triage time. Interrater agreement was estimated with κ statistics with 95% confidence intervals (CIs).
Results:
A total of 1,491 individual triage assessments (752 pre-eCTAS, 739 post-implementation) were audited during 42 7-hour triage shifts (21 pre-eCTAS, 21 post-implementation). Exact modal agreement was achieved for 567 patients (75.4%) pre-eCTAS compared with 685 patients (92.7%) triaged with eCTAS. With the auditor's CTAS score as the reference, eCTAS significantly reduced the number of patients over-triaged (12.0% versus 5.1%; Δ 6.9; 95% CI 4.0 to 9.7) and under-triaged (12.6% versus 2.2%; Δ 10.4; 95% CI 7.9 to 13.2). Interrater agreement was higher with eCTAS (unweighted κ 0.89 versus 0.63; quadratic-weighted κ 0.93 versus 0.79). Median triage time was 312 seconds (n=3,808 patients) pre-eCTAS and 347 seconds (n=3,489 patients) with eCTAS (Δ 35 seconds; 95% CI 29 to 40 seconds).
Conclusion:
A standardized, electronic approach to performing triage assessments improves both interrater agreement and data accuracy without substantially increasing triage time.
