Related Experiment Videos
Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model
Stephen L Hillis1, Kevin S Berbaum
1Center for Research in the Implementation of Innovative Strategies in Practice (CRIISP), Iowa City VA Medical Center (152), 601 Highway 6 West, Iowa City, IA 52246-2208, USA. steve-hillis@uiowa.edu
Academic Radiology
|December 3, 2005
Summary
The Dorfman-Berbaum-Metz (DBM) method for ROC studies performs better with normalized pseudovalues and less data-based model simplification. These modifications improve AUC estimates and reduce errors in multireader ROC analysis.
Area of Science:
- Medical Imaging Analysis
- Statistical Methods in Medicine
- Receiver Operating Characteristic (ROC) Analysis
Background:
- The Dorfman-Berbaum-Metz (DBM) method, used for multireader ROC studies, can be overly conservative and yield inaccurate AUC estimates outside the valid parameter space.
- Previous research suggested using normalized pseudovalues to correct AUC estimates and employing less data-based model simplification to address DBM method limitations.
Purpose of the Study:
- To empirically evaluate the performance improvements of the DBM method when incorporating two key modifications: normalized pseudovalues and reduced data-based model simplification.
- To assess if these combined modifications enhance the accuracy and reliability of AUC estimates in multireader ROC analysis.
Main Methods:
- A comprehensive simulation study was conducted, examining the DBM procedure with both proposed modifications for discrete and continuous rating data.
- The study involved 144 distinct combinations of reader/case sample sizes, case ratios, and variance components to compare modalities based on ROC area.
- Both parametric and nonparametric estimation methods were utilized for ROC area estimation.
Main Results:
- The DBM procedure incorporating both normalized pseudovalues and less data-based model simplification demonstrated superior performance compared to the original DBM or versions with only one modification.
- For parametric estimation with discrete ratings, the dual modification approach yielded a mean Type I error rate (0.043) closest to the nominal 0.05 level.
- This approach also resulted in the smallest range (0.050) and standard deviation (0.0108) across the 144 evaluated Type I error rates.
Conclusions:
- The study recommends the adoption of normalized pseudovalues along with less data-based model simplification when utilizing the DBM procedure for ROC analysis.
- These modifications are shown to enhance the performance and reliability of the DBM method in multireader ROC studies.