Related Experiment Video
Updated: Jul 5, 2026

Doppler Ultrasound-Based Leg Blood Flow Assessment During Single-Leg Knee-Extensor Exercise in an Uncontrolled Setting
Published on: December 15, 2023
Reliability studies of diagnostic tests are not using enough observers for robust estimation of interobserver
Mohsen Sadatsafavi1, Mehdi Najafzadeh, Larry Lynd
1Center for Health Evaluation and Outcome Research (CHEOS), University of British Columbia, Vancouver BC, Canada V6Z 1Y6. msafavi@interchange.ubc.ca
Objective:
Any attempt to generalize the performance of a subjective diagnostic method should take into account the sample variation in both cases and readers. Most current measures of the performance of a test, especially the indices of reliability, only tackle the variation of cases, and hence are not suitable for generalizing results across the population of readers. We attempted to study the effect of readers' variation on two measures of multireader reliability: pair-wise agreement and Fleiss' kappa.
Study Design And Setting:
We used a normal hierarchical model with a latent trait (signal) variable to simulate a binary decision-making task by different number of readers on an infinite sample of cases.
Results:
It could be shown that both measures, especially Fleiss' kappa, have a large sample variance when estimated by a small number of readers, casting doubt on their accuracy given the number of readers typically used in current reliability studies.
Conclusion:
The majority of the current agreement studies is likely limited by the number of readers and is unlikely to produce a reliable estimate of reader agreement.
Related Concept Videos
Reliability and Validity
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
Accuracy and Errors in Hypothesis Testing
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5% chance...
Systematic Error: Methodological and Sampling Errors
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Naturalistic Observations
Uncertainty in Measurement: Accuracy and Precision
