Estimation and comparison of CAD system performance in clinical settings

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Biostatistics

Background:

• Computer-aided detection (CAD) systems are crucial in medical diagnostics.
• Comparing CAD systems often involves free-response receiver operating characteristic (FROC) curves.
• Accurate comparison requires determining the system operating point and accounting for sampling error.

Purpose of the Study:

• To present a method for capturing the effect of sampling error on determining the correct CAD operating point.
• To estimate the probability of one CAD system outperforming another in a clinical setting.
• To assess the impact of training set size on CAD system performance comparison.

Main Methods:

• Examined the distribution of clinical outcomes from two artificial CAD systems with differing FROC curves.
• Captured sampling error through the distribution of system thresholds for a specified sensitivity.
• Introduced a measure of superiority to determine the probability of one system being better than another.

Main Results:

• For mammography CAD systems trained on 100 cases, FROC curves need a separation of 0.20 false positives per image for a 90% probability of superiority.
• Increasing training set size beyond 100 cases showed no apparent performance gain.
• The method accounts for uncertainty in determining the CAD operating point.

Conclusions:

• The presented method is applicable to any computer-aided detection system evaluated with FROC curves.
• It enables the construction of confidence intervals for clinical outcomes.
• The method highlights the importance of FROC curve separation and training set size in system comparison.