Sources of performance variability in deep learning-based polyp detection

Area of Science:

• Medical Imaging
• Computer Vision
• Gastroenterology

Background:

• Reliable validation metrics are crucial for scientific progress and clinical translation of medical imaging methods.
• Existing theoretical frameworks for metric pitfalls lack experimental validation in specific applications.
• Colon cancer screening using deep learning requires robust evaluation strategies.

Purpose of the Study:

• To address the gap in experimental evidence regarding validation metric pitfalls in colon cancer screening.
• To present the winning solution of the Endoscopy Computer Vision Challenge for colon cancer detection.
• To demonstrate the sensitivity of common metrics to hyperparameters and the impact of poor metric choices.

Main Methods:

• Analysis of patient data from six clinical centers.
• Evaluation of commonly applied object detection metrics.
• Testing standard computer vision hyperparameters for clinical plausibility.
• Development of localization criteria for clinical relevance.

Main Results:

• Commonly applied object detection metrics exhibit high inter-center variability.
• Standard computer vision hyperparameters do not consistently yield clinically plausible results.
• New localization criteria demonstrate good correlation with clinical relevance.

Conclusions:

• Performance results in polyp detection are highly sensitive to design choices.
• Current metric configurations often fail to meet clinical needs due to suboptimal hyperparameters.
• Comparing performance across datasets can be misleading, highlighting the need for revised validation strategies in deep learning-based colonoscopy.