Related Experiment Video
Updated: Mar 16, 2026
07:28
A Protocol of Manual Tests to Measure Sensation and Pain in Humans
Published on: December 19, 2016
21.8K
Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate?
Antonia Zapf1, Stefanie Castell2, Lars Morawietz3
1Department of Medical Statistics, University Medical Center Göttingen, Humboldtallee 32, 37073, Göttingen, Germany. Antonia.Zapf@med.uni-goettingen.de.
BMC Medical Research Methodology
|August 7, 2016
Summary
Fleiss' kappa and Krippendorff's alpha with bootstrap confidence intervals are suitable for nominal data reliability. Krippendorff's alpha is recommended for missing data or higher-order data, and the asymptotic confidence interval for Fleiss' kappa should be avoided.
Area of Science:
- Medical research methodology
- Statistical analysis in healthcare
Background:
- Measurement reliability is crucial for medical research.
- Fleiss' kappa (Fleiss' K) and Krippendorff's alpha are flexible measures for nominal data inter-rater reliability.
- Investigating statistical properties of these measures and confidence intervals is essential.
Purpose of the Study:
- To compare Fleiss' K and Krippendorff's alpha for inter-rater reliability.
- To evaluate precision of estimates and coverage probability of confidence intervals.
- To determine suitability in various data scenarios.
Main Methods:
- Large-scale simulation study.
- Investigated precision of Fleiss' K and Krippendorff's alpha estimates.
- Assessed empirical coverage probability of asymptotic and bootstrap confidence intervals.
Main Results:
- Fleiss' K and Krippendorff's alpha point estimates were similar for complete data.
- Krippendorff's alpha provided stable estimates with missing data; Fleiss' K yielded biased estimates.
- Bootstrap confidence intervals showed accurate coverage, unlike the asymptotic interval for Fleiss' K.
Conclusions:
- Fleiss' K and Krippendorff's alpha with bootstrap CIs are suitable for complete nominal data.
- Avoid asymptotic confidence intervals for Fleiss' K.
- Krippendorff's alpha is recommended for missing data or higher-order categories.
Related Concept Videos
01:20Kendall's Coefficient of Concordance
1.1K
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects...
1.1K
00:54Ratio Level of Measurement
22.1K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
22.1K
00:56Nominal Level of Measurement
41.3K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. Not every statistical operation can be used with every set of data. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
The data that cannot be measured but can be grouped into categories fall under the nominal level of measurement. Data that is measured using a nominal...
The data that cannot be measured but can be grouped into categories fall under the nominal level of measurement. Data that is measured using a nominal...
41.3K
01:12Sign Test for Nominal Data
429
The sign test is a nonparametric method used to evaluate hypotheses about the median of a single sample or to compare the medians of two related samples. The sign test is particularly useful when dealing with nominal data, which includes distinct categories without an inherent order, such as names, labels, and preferences. Nominal data restricts statistical analysis to evaluating population proportions rather than mean or median values that require continuous data.
For example, consider a...
For example, consider a...
429
01:23McNemar's Test
954
McNemar's Test is a nonparametric statistical test used to determine if there is a significant difference in proportions between two related groups when the outcome is binary (e.g., yes/no, success/failure). It is beneficial when we have paired data, such as pre-test/post-test designs, where the same subjects are measured under two different conditions. The test is named after the statistician Quinn McNemar, who introduced it in 1947. It is commonly used in situations where subjects are...
954
00:55Ordinal Level of Measurement
36.4K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
Data measured using an ordinal scale are similar to nominal scale data, but there is one major difference. The ordinal scale data can be ordered. An example of ordinal scale data is a list of the top five national parks...
Data measured using an ordinal scale are similar to nominal scale data, but there is one major difference. The ordinal scale data can be ordered. An example of ordinal scale data is a list of the top five national parks...
36.4K