Related Experiment Video
Updated: Jun 4, 2025
07:28
A Protocol of Manual Tests to Measure Sensation and Pain in Humans
Published on: December 19, 2016
20.9K
A comprehensive guide to study the agreement and reliability of multi-observer ordinal data
Sophie Vanbelle1, Christina Hernandez Engelhart2,3, Ellen Blix3
1Methodology and Statistics, CAPHRI, Maastricht University, P. Debyeplein, 1, Maastricht, 6229 HA, The Netherlands. sophie.vanbelle@maastrichtuniversity.nl.
BMC Medical Research Methodology
|December 20, 2024
Summary
This study provides guidance on selecting and planning agreement and reliability studies for ordinal outcomes with multiple observers. It introduces methods and open-source software to improve the quality of clinical test reliability research.
Area of Science:
- Statistics
- Biostatistics
- Psychometrics
Background:
- Systematic reviews highlight issues in agreement and reliability studies for ordinal scales, particularly with multiple observers.
- Existing guidelines need enhancement for comprehensive reporting and execution of such studies.
Purpose of the Study:
- To provide comprehensive information on selecting and planning agreement and reliability studies for ordinal outcomes.
- To address challenges in studies involving more than two observers.
Main Methods:
- Generalization of agreement measures (proportion of agreement/disagreement, mean absolute deviation, mean squared deviation, weighted kappa) for ordinal outcomes with multiple observers.
- Utilizing the delta method for large sample variance estimation to construct Wald confidence intervals.
- Developing a procedure to determine the minimum number of raters and patients required to minimize sampling uncertainty.
Main Results:
- Clear interpretation of agreement and reliability coefficients is provided, differentiating between the two concepts.
- Sample variance for various coefficients is presented or derived, enabling Wald confidence interval construction.
- A procedure for sample size determination (raters and patients) is established to control uncertainty.
Conclusions:
- The paper complements existing guidelines like GRRAS to enhance the quality of reliability and agreement studies for clinical tests.
- Open-source software (R package and Shiny application) is provided to facilitate the application of these methods for researchers.
Keywords:
AgreementClinical test validationGuidelineMeasurement errorReliabilityRepeatabilityReproducibilityMore Related Videos
Related Concept Videos
00:55Ordinal Level of Measurement
22.9K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
Data measured using an ordinal scale are similar to nominal scale data, but there is one major difference. The ordinal scale data can be ordered. An example of ordinal scale data is a list of the top five national parks...
Data measured using an ordinal scale are similar to nominal scale data, but there is one major difference. The ordinal scale data can be ordered. An example of ordinal scale data is a list of the top five national parks...
22.9K
01:29Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
01:20Kendall's Coefficient of Concordance
246
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects...
246
01:21Friedman Two-way Analysis of Variance by Ranks
144
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
144
01:11Statistical Analysis: Overview
6.1K
When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
6.1K
02:30Naturalistic Observations
15.4K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
15.4K