MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images

Area of Science:

• Medical Imaging
• Machine Learning
• Artificial Intelligence in Medicine

Background:

• Out-of-Distribution (OoD) detection is vital for safe AI deployment in medicine, as algorithms often err on unseen data.
• Existing benchmarks for OoD detection lack focus on the critical medical imaging domain.
• Accurate OoD detection can aid clinicians in identifying incidental findings.

Purpose of the Study:

• To establish the Medical-Out-Of-Distribution-Analysis-Challenge (MOOD) as an open, fair, and unbiased benchmark for OoD methods in medical imaging.
• To evaluate the performance of current OoD detection algorithms on medical imaging data.
• To identify challenges and potential improvements for OoD detection in clinical settings.

Main Methods:

• Introduction of the MOOD challenge, a novel benchmark for Out-of-Distribution detection in medical imaging.
• Analysis of algorithms submitted to the MOOD challenge.
• Evaluation of algorithm performance against perceived difficulty and anomaly types.
• Assessment of correlation between challenge ranking and performance on a toy test set.

Main Results:

• Algorithm performance strongly correlates with the perceived difficulty of Out-of-Distribution cases.
• All evaluated algorithms exhibit high variance in performance across different types of anomalies.
• Current OoD detection algorithms are not yet consistently reliable for clinical practice.
• A strong correlation exists between challenge ranking and performance on a simplified toy dataset.

Conclusions:

• The MOOD benchmark provides a critical resource for advancing Out-of-Distribution detection in medical imaging.
• Significant challenges remain in developing robust and reliable OoD detection methods for clinical use.
• A simple toy test set may serve as a useful proxy for algorithm development and initial evaluation.