Objective evaluation of deep uncertainty predictions for COVID-19 detection

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Machine Learning for Diagnostics

Background:

• Deep neural networks (DNNs) are used for COVID-19 detection in medical images, but their generalization is limited by small datasets and lack of predictive confidence.
• Quantifying prediction uncertainties is crucial for the reliable deployment of DNNs in clinical settings.

Purpose of the Study:

• To apply and evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-ray (CXR) images.
• To introduce a novel uncertainty confusion matrix and new performance metrics for evaluating uncertainty estimates objectively.

Main Methods:

• Application and comparative evaluation of three uncertainty quantification techniques for DNNs.
• Development of an uncertainty confusion matrix and novel performance metrics for uncertainty estimation.
• Experiments using chest X-ray images for COVID-19 detection.

Main Results:

• DNNs trained on CXR images outperform those trained on natural image datasets like ImageNet.
• Predictive uncertainty estimates are significantly higher for incorrect predictions than correct ones.
• Ensemble methods demonstrate more reliable uncertainty capture during inference.

Conclusions:

• Uncertainty quantification methods can effectively flag high-risk, erroneous DNN predictions for COVID-19 detection.
• The proposed uncertainty evaluation metrics provide a quantitative measure of trust in DNN predictions for CXR-based COVID-19 diagnosis.
• The novel metrics are generalizable for evaluating probabilistic forecasts in various classification tasks.