Related Experiment Video
Updated: Jan 6, 2026

Author Spotlight: Assessing the Reliability of Doppler Ultrasound in Measuring Leg Blood Flow
Published on: December 15, 2023
Sequentially Determined Measures of Interobserver Agreement (Kappa) in Clinical Trials May Vary Independent of
Russell Reeve1, Klaus Gottlieb2
1Decision Sciences, IQVIA, Durham, NC, USA.
Background:
Cohen's kappa is a statistic that estimates interobserver agreement. It was originally introduced to help develop diagnostic tests. Interpretative readings of 2 observers, for example, of a mammogram or other imaging, were compared at a single point in time. It is known that kappa depends on the prevalence of disease and that, therefore, kappas across different settings are hard to compare.
Methods:
Using simulation, we examine an analogous situation, not previously described, that occurs in clinical trials where sequential measurements are obtained to evaluate disease progression or clinical improvement over time.
Results:
We show that weighted kappa, used for multilevel outcomes, changes during the trial even if we keep the performance of the observer constant.
Conclusions:
Kappa and closely related measures can therefore only be used with great difficulty, if at all, in quality assurance in clinical trials.
More Related Videos
08:40Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
Published on: June 12, 2019
08:52Software-Assisted Quantitative Measurement of Osteoarthritic Subchondral Bone Thickness
Published on: March 18, 2022
Related Concept Videos
Kendall's Coefficient of Concordance
Variability: Analysis
The range is a simple measure of variability, indicating the difference between the highest and...
Bioequivalence Data: Statistical Interpretation
Drug Product Performance: In Vitro–In Vivo Correlation
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...