Related Experiment Video
Updated: Jun 13, 2026

Computerized Adaptive Testing System of Functional Assessment of Stroke
Published on: January 7, 2019
Calculating kappas from adjusted data improved the comparability of the reliability of triage systems: a comparative
Ineke van der Wulp1, Henk F van Stel
1Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands. i.vanderwulp@umcutrecht.nl
Objectives:
It is difficult to compare the reliability of triage systems with the kappa statistic. In this article, a method for comparing triage systems was developed and applied to previously conducted triage reliability studies.
Study Design And Setting:
From simulations with theoretical distributions, the minimum, normal, and maximum weighted kappa for 3- to 5-level triage systems were computed. To compare the reliability of triage systems in previously conducted triage reliability studies, the normal kappa was calculated. Furthermore, the reported quadratically weighted kappas were compared with the minimum, normal, and maximum weighted kappa to characterize the degree and direction of skewness of the data.
Results:
The normal kappa was higher in 3-level triage systems (median: κ=0.84) compared with 4-level (median: κ=0.37) and 5-level (median: κ=0.57) systems. In 3-level triage systems, the percentages observed agreement were unequally distributed, which resulted in small quadratically weighted kappas. In 4- and 5-level systems, the percentages observed agreement were more equally distributed compared with 3-level systems, which resulted in higher quadratically weighted kappa values.
Conclusion:
When comparing triage systems with different numbers of categories, one should report both the normal and quadratically weighted kappa. Calculating normal kappas from previously conducted triage reliability studies revealed substantial theoretical differences in interrater reliability of triage systems than previously reported.
Related Concept Videos
Kaplan-Meier Approach
Kendall's Tau Test
A τ value of +1 indicates that...
Comparing the Survival Analysis of Two or More Groups
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Actuarial Approach
Consider the example of a high-risk surgical procedure with significant early-stage mortality. A two-year clinical study is conducted,...
One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
On...