Inter-rater reliability of a computerized presenting-complaint-linked triage system in an urban emergency department

Triage reliability studies typically use hypothetical scenarios and weighted kappa scores where agreement within one level is considered satisfactory. But if triage category is used to help define ED case-mix groups for comparative or benchmarking processes, agreement on exact triage level and major system involved is important. Our hypothesis was that a computerized menu that links presenting complaints to preferred triage levels (PC-linked triage) would provide high triage reliability.

Our objective was to assess inter-rater reliability of PC-linked triage using the Canadian Emergency Department Triage and Acuity Scale (CTAS) in a real-time clinical setting, considering agreement on exact triage level and primary body system involved.

On duty triage nurses entered patient presenting complaint and PC-linked triage level as per standard procedure. In a convenience sample of patients, a second nurse, blinded to triage assignment, observed the triage interaction and independently entered presenting complaint and triage level on a dummy terminal.

During the study, 15 nurse pairs triaged 266 patients. Study patients matched actual emergency department case mix closely. Triage nurses agreed exactly in 74% of cases and within one level in 94% of cases. The unweighted kappa value was 0.66 (95% confidence interval [CI], 0.60-0.73) and the quadratic weighted kappa value was 0.75 (95% CI, 0.68-0.81). Kappa for agreement on major system involved was 0.80 (95% CI, 0.69-0.91).

PC-linked triage has high inter-rater reliability in a real-time clinical setting. PC-linked triage may be useful as one factor in defining case-mix groups for benchmarking and comparative purposes.