Related Experiment Video
Updated: Jun 23, 2026
Inverse Probability of Treatment Weighting (Propensity Score) using the Military Health System Data Repository and National Death Index
Published on: January 8, 2020
Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage
Ineke van der Wulp1, Henk F van Stel
1Department of Public Health, Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht, The Netherlands. I.vanderwulp@umcutrecht.nl
Objective:
Mistriage can have serious consequences for patients with urgent complaints. We therefore reviewed the assessment of triage-reliability, and propose an alternative weighted kappa that accounts for severity of mistriage.
Study Design And Setting:
A systematic literature search was performed in three online databases and a search engine. An alternative kappa weighting scheme was developed (triage-weighted kappa), and kappas of previous conducted reliability studies were recalculated.
Results:
Kappa is the most frequently used statistic in triage-reliability studies (n=33). More than half of the studies did not report the type of kappa that was used. Linear and quadratically weighted kappa values do not reflect the seriousness of mistriage. Several studies reported almost perfect agreement, whereas percentages of mistriage ranged between 11.1% and 43.4%. In all studies, triage-weighted kappa was lower than reported kappas with a mean difference of 0.17 (range: 0.04-0.32).
Conclusion:
No existing studies on reliability of triage systems account for mistriage. Using triage-weighted kappa, which reflects severity of mistriage, shows that the reliability of triage systems is lower than reported.
Insights
Triage reliability studies often overlook mistriage severity. A new triage-weighted kappa reveals lower system reliability than previously reported, highlighting critical patient safety concerns.
Area of Science:
- Emergency Medicine
- Biostatistics
- Healthcare Quality
Background:
- Patient mistriage during emergency department (ED) triage can lead to severe adverse outcomes.
- Assessing the reliability of triage systems is crucial for patient safety.
- Existing statistical methods may not adequately capture the impact of mistriage severity.
Purpose of the Study:
- To systematically review the assessment of triage reliability in existing literature.
- To propose and validate an alternative weighted kappa statistic that incorporates the severity of mistriage.
- To re-evaluate the reliability of triage systems using the proposed method.
Main Methods:
- A comprehensive systematic literature search was conducted across multiple databases and a search engine.
- A novel weighting scheme, termed triage-weighted kappa, was developed to account for mistriage severity.
- Kappa values from previously published triage reliability studies were recalculated using the new method.
Main Results:
- Kappa statistics are widely used in triage reliability research, but the specific type is often unreported.
- Standard weighted kappa methods fail to reflect the clinical seriousness of mistriage.
- Recalculation using triage-weighted kappa consistently yielded lower reliability scores compared to reported values, with a mean difference of 0.17.
Conclusions:
- Current studies on triage system reliability do not adequately account for the severity of patient mistriage.
- The proposed triage-weighted kappa offers a more accurate assessment of triage reliability by considering mistriage severity.
- Implementing triage-weighted kappa demonstrates that the actual reliability of many triage systems is lower than previously understood, emphasizing the need for improved accuracy.
