Role of calibration in uncertainty-based referral for deep learning

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Ophthalmology

Background:

• Assessing prediction uncertainty in deep neural networks (DNNs) for medical imaging is critical for clinical decision-making.
• Diabetic retinopathy detection using DNNs highlights the need for reliable uncertainty quantification.

Purpose of the Study:

• To empirically evaluate the impact of model calibration on uncertainty-based referral strategies.
• To investigate how different DNN configurations and training parameters affect referral effectiveness.

Main Methods:

• Utilized diabetic retinopathy detection data for analysis.
• Compared various network architectures, uncertainty estimation methods, and training data sizes.
• Assessed the relationship between model calibration and the performance of uncertainty-based referral.

Main Results:

• A strong correlation was found between effective uncertainty-based referral and well-calibrated DNN models.
• Complex DNNs often exhibit significant calibration errors, impacting referral accuracy.
• Post-calibration of DNNs enhanced the ability of uncertainty-based referral to identify difficult-to-classify observations.

Conclusions:

• Model calibration is a key factor for successful uncertainty-based referral in medical AI.
• Addressing calibration errors in DNNs is essential for reliable clinical implementation.
• Post-calibration offers a viable method to improve decision support systems for conditions like diabetic retinopathy.