Related Experiment Video
Updated: Feb 13, 2026

Comparative Lesions Analysis Through a Targeted Sequencing Approach
Published on: November 5, 2019
The Problems with the Kappa Statistic as a Metric of Interobserver Agreement on Lesion Detection Using a Third-reader
Joanna H Shih1, Matthew D Greer2, Baris Turkbey2
1Biometric Research Program, NCI, NIH, 9609 Medical Center Drive, Room 5W124, MSC 9735, Bethesda, MD 20892.
Rationale And Objectives:
To point out the problems with Cohen kappa statistic and to explore alternative metrics to determine interobserver agreement on lesion detection when locations are not prespecified.
Materials And Methods:
Use of kappa and two alternative methods, namely index of specific agreement (ISA) and modified kappa, for measuring interobserver agreement on the location of detected lesions are presented. These indices of agreement are illustrated by application to a retrospective multireader study in which nine readers detected and scored prostate cancer lesions in 163 consecutive patients (n = 110 cases, n = 53 controls) using the guideline of Prostate Imaging Reporting and Data System version 2 on multiparametric magnetic resonance imaging.
Results:
The proposed modified kappa, which properly corrects for the amount of agreement by chance, is shown to be approximately equivalent to the ISA. In the prostate cancer data, average kappa, modified kappa, and ISA equaled 30%, 55%, and 57%, respectively, for all lesions and 20%, 87%, and 87%, respectively, for index lesions.
Conclusions:
The application of kappa could result in a substantial downward bias in reader agreement on lesion detection when locations are not prespecified. ISA is recommended for assessment of reader agreement on lesion detection.
Related Concept Videos
Statistical Significance
Probability in Statistics
An example of a simple event is a coin toss. The result of a coin toss is either a head or a tail. Here, head and tail are two simple events. These two simple events make up the sample space. Further, the probability of an event occurring falls within the range of 0 to 1. The probability of an...
Introduction to Statistics
In statistics, the collection of individuals or objects under study is called population. The idea of sampling is to select a portion of the larger population...
Location and Orientation of the Heart
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
Identifying Statistically Significant Differences: The F-Test

