The empirical variance estimator for computer aided diagnosis: lessons for algorithm validation.

Area of Science:

• Medical Image Analysis
• Computational Pathology
• Biostatistics

Background:

• Computer-aided diagnosis (CAD) pipelines require reliable performance comparison for refinement.
• Variance estimation is critical for reliable comparison but faces challenges with small sample sizes in supervised learning.
• Current methods often exhibit bias, hindering accurate algorithm assessment.

Purpose of the Study:

• To propose and validate a novel empirical variance estimator for medical image analysis.
• To address the limitations of existing variance estimation methods in the context of small datasets.
• To improve the statistical rigor of performance comparisons in computer-aided diagnosis.

Main Methods:

• An empirical variance estimator based on validation within disjoint data subsets was developed.
• Resampling experiments were conducted using Alzheimer's disease classification on the ADNI dataset.
• The proposed estimator's behavior was compared against naive approaches.

Main Results:

• The proposed variance estimator was demonstrated to be unbiased.
• The new estimator provided more accurate estimates compared to naive methods, which were biased downwards.
• The estimator accommodates arbitrary validation strategies and performance metrics without independence assumptions.

Conclusions:

• The proposed unbiased empirical variance estimator enhances the reliability of algorithm performance comparisons in medical image analysis.
• This method facilitates statistically convincing comparisons and confidence intervals, crucial for pipeline development.
• The estimator can guide the selection of optimal validation strategies for CAD systems.