Emergency medical services triage using the emergency severity index: is it reliable and valid?

Efficient communication between emergency medical services (EMS) and ED providers using a common triage system may enable more effective transfers when EMS arrives in the emergency department. We sought (1) to evaluate inter-rater reliability between Emergency Severity Index (ESI) assignments designated by EMS personnel and emergency triage nurses (registered nurses [RNs]) and (2) to evaluate the validity of EMS triage assignments using the ESI instrument.

This prospective, observational study evaluated inter-rater reliability in ESI scores assigned by prehospital personnel and RNs. EMS providers were trained to use the ESI by the same methods used for nurse training. EMS personnel assigned triage scores to patients independent of assignments by the RN. Inter-rater reliability, differences based on provider experience, and validity of EMS triage assignments (sensitivity and specificity) were evaluated.

Seventy-five paired, blinded triages were completed. Overall concordance between EMS providers and RNs was 0.409 (95% confidence interval [CI], 0.256-0.562). Agreement for EMS providers with less experience was 0.519 (95% CI, 0.258-0.780), whereas concordance for those with more experience was 0.348 (95% CI, 0.160-0.536; χ(2) = 1.413, df = 1, P = .235). Sensitivity ranged from 0% to 67.86%. Specificity ranged from 68.09% to 97.26%.

We observed moderate concordance between EMS and RN ESI triage assignments. EMS sensitivity for correct acuity assignment was generally poor, whereas specificity for correctly not assigning a particular level was better. Additional research investigating the potential causes of the poor agreement that we observed is warranted.