Related Experiment Video
Updated: Sep 10, 2026
Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
Published on: June 12, 2019
Interrater Agreement Between Paramedic CTAS Scores and an Electronic CTAS Tool: A Retrospective Study Across Eastern
Michael Slatter1, Pria Nippak1, Nadia Nandlall1
1School of Health Services Management, Ted Rogers School of Management, Toronto Metropolitan University, Toronto, Canada, ryerson.ca.
Objectives:
To assess interrater agreement between the Canadian triage and acuity scale (CTAS) scores assigned by paramedics and those assigned by a paramedic rater using an electronic smartphone application, aiming to identify training gaps and potential patient safety risks associated with CTAS scoring in Canadian hospital emergency departments (CEDs).
Methods:
A retrospective analysis was conducted using 192,326 CTAS-level call data collected from electronic patient care records of nine paramedic services. Data were entered into an electronic CTAS smartphone app. The calls were categorized based on contact level and the sample size. Minitab software and VassarStats were used to calculate the quadratic-weighted Cohen's kappa (k) coefficient.
Results:
A retrospective analysis of 1651 paramedic call reports from across Eastern Ontario compared paramedic-assigned CTAS scores with those generated using an eCTAS smartphone tool. Overall agreement between raters was substantial (k = 0.78), with 55.24% (95% CI: 52.8, 57.66) agreement for CTAS on contact and 52.88% (95% CI: 50.44, 55.31) for CTAS on departure (k = 0.76). Paramedics assigned more low-acuity CTAS 4-5 scores than the eCTAS tool on both contact (39% vs. 31%) and departure (40% vs. 31%). Additionally, 11%-12% of CTAS 2 calls were downgraded by paramedics to CTAS 4 or 5, compared with 7% when eCTAS was used. Cohen's kappa demonstrated near-perfect agreement for CTAS 1 scores, fair agreement for CTAS 2 and 3, slight agreement for CTAS 4, and moderate agreement for CTAS 5 categories. Complaint-specific analyses showed the greatest disagreement in the gastrointestinal and substance overdose/misuse categories.
Conclusions:
This study found near-perfect agreement between paramedic documentation and the paramedic smartphone application for CTAS-1 in acute patients, with moderate agreement for other levels. Further studies should include multiple raters to better understand the implications of using a smartphone application.
