Predictive uncertainty estimation for out-of-distribution detection in digital pathology

Area of Science:

• Medical Imaging
• Machine Learning
• Artificial Intelligence in Healthcare

Background:

• Clinical deployment of machine learning (ML) models requires robust uncertainty estimation for real-time risk assessment.
• Accurate models must identify uncertainty for unseen data and abnormalities, crucial for patient safety.
• Existing uncertainty estimation methods lack rigorous evaluation on large-scale digital pathology datasets.

Purpose of the Study:

• To benchmark prevalent uncertainty estimation methods on diverse digital pathology datasets.
• To evaluate method performance on in-distribution and realistic out-of-distribution (OOD) data at the whole-slide level.
• To compare patch-based uncertainty aggregation with whole-slide level scores.

Main Methods:

• Aggregated uncertainty values from patch-based classifiers to derive whole-slide level uncertainty scores.
• Evaluated uncertainty estimation on multiple digital pathology datasets, including near and far OOD data.
• Compared deep ensembles and multi-head ensembles for OOD detection performance.

Main Results:

• Classical computer vision benchmark results do not consistently translate to medical imaging.
• Deep ensembles perform best for far-OOD detection; multi-head ensembles excel at near-OOD detection.
• Out-of-distribution data significantly harms deployed ML model performance.
• Uncertainty estimates effectively discriminate in-distribution from OOD data (high AUC).

Conclusions:

• Uncertainty estimation is vital for safe ML deployment in clinical settings, particularly in digital pathology.
• Ensemble methods show promise for OOD detection, but optimal choice depends on the OOD data type (near vs. far).
• Model deployment necessitates careful tuning informed by potential OOD data characteristics to ensure reliability.