Related Experiment Video
Updated: Feb 25, 2026
Meta-analysis of Voxel-Based Neuroimaging Studies using Seed-based d Mapping with Permutation of Subject Images SDM-PSI
Published on: November 27, 2019
MRMC analysis of agreement studies
Brandon D Gallas1, Amrita Anam1,2, Weijie Chen1
1CDRH/OSEL Division of Imaging, Diagnostics, and Software Reliability, 10903 New Hampshire Ave, Silver Spring, MD, 20993.
Abstract:
The purpose of this work is to present and evaluate methods based on U-statistics to compare intra- or inter-reader agreement across different imaging modalities. We apply these methods to multi-reader multi-case (MRMC) studies. We measure reader-averaged agreement and estimate its variance accounting for the variability from readers and cases (an MRMC analysis). In our application, pathologists (readers) evaluate patient tissue mounted on glass slides (cases) in two ways. They evaluate the slides on a microscope (reference modality) and they evaluate digital scans of the slides on a computer display (new modality). In the current work, we consider concordance as the agreement measure, but many of the concepts outlined here apply to other agreement measures. Concordance is the probability that two readers rank two cases in the same order. Concordance can be estimated with a U-statistic and thus it has some nice properties: it is unbiased, asymptotically normal, and its variance is given by an explicit formula. Another property of a U-statistic is that it is symmetric in its inputs; it doesn't matter which reader is listed first or which case is listed first, the result is the same. Using this property and a few tricks while building the U-statistic kernel for concordance, we get a mathematically tractable problem and efficient software. Simulations show that our variance and covariance estimates are unbiased.
Insights
This study introduces U-statistic methods for comparing reader agreement in multi-reader multi-case studies. These novel techniques accurately estimate agreement variance across imaging modalities, improving diagnostic accuracy.
Area of Science:
- Medical Imaging
- Biostatistics
- Pathology
Background:
- Evaluating reader agreement is crucial for diagnostic accuracy in medical imaging.
- Traditional methods may not fully account for variability in multi-reader multi-case (MRMC) studies.
- Comparing agreement across different imaging modalities (e.g., microscopy vs. digital scans) presents unique challenges.
Purpose of the Study:
- To present and evaluate U-statistic-based methods for assessing intra- and inter-reader agreement.
- To apply these methods to multi-reader multi-case (MRMC) studies, specifically in pathology.
- To estimate reader-averaged agreement and its variance, considering reader and case variability.
Main Methods:
- Utilized U-statistics to estimate concordance, a measure of agreement defined as the probability of two readers ranking two cases similarly.
- Developed methods to estimate the variance of reader-averaged agreement within an MRMC framework.
- Leveraged the symmetric properties of U-statistics for mathematical tractability and efficient software development.
Main Results:
- U-statistic methods provide unbiased estimates for concordance.
- Variance and covariance estimates for reader agreement were shown to be unbiased through simulations.
- The developed methods efficiently handle variability from both readers and cases in MRMC analyses.
Conclusions:
- U-statistic-based methods offer a robust and statistically sound approach for comparing reader agreement across imaging modalities.
- These methods are particularly valuable for complex MRMC studies in fields like pathology.
- The proposed techniques enhance the reliability of agreement assessments in diagnostic imaging.
Related Concept Videos
01:13Multiple Comparison Tests
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
01:23McNemar's Test
01:52Regression Toward the Mean
01:13¹H NMR: Complex Splitting
Splitting diagrams or splitting tree diagrams are routinely used to depict such complex couplings. While drawing splitting diagrams, the splitting with the larger coupling constant is usually applied...