A method for measuring interrater agreement on checklists
Related Experiment Video
Updated: Oct 18, 2026

Nest Building Behavior as an Early Indicator of Behavioral Deficits in Mice
Published on: October 19, 2019
J M Sinacore1, K J Connell, A J Olthoff
1University of Illinois at Chicago, Department of Family Medicine 60612-7248, USA. Sinacore@uic.edu
Abstract:
A method for measuring interrater agreement on checklists is presented. This technique does not assign individual scores to raters, but computes a single agreement score from the concordance of their check mark configurations. An overall coefficient of agreement, called phi, is derived. The agreement coefficient that is expected by chance and the statistical significance of phi are determined by statistical simulation. Despite the dichotomous nature of the checklist agreement (raters either agree or disagree on items), we show that the binomial distribution does not provide a means for testing the statistical significance of phi. A medical education study is used to illustrate the phi methodology.
Related Concept Videos
Kendall's Coefficient of Concordance
Ratio Level of Measurement
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated. For...
Multiple Comparison Tests
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
Ordinal Level of Measurement
Data measured using an ordinal scale are similar to nominal scale data, but there is one major difference. The ordinal scale data can be ordered. An example of ordinal scale data is a list of the top five national parks in the...
Behrens–Fisher Test
This test is...
Spearman's Rank Correlation Test
Spearman's test calculates correlation by...
