Evolutionary Strategies Enable Systematic and Reliable Uncertainty Quantification: A Proof-of-Concept Pilot Study on
Joseph N Stember1, Katharine Dishner2, Mehrnaz Jenabi2
1Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, 10021, USA. joestember@gmail.com.
Journal of Imaging Informatics in Medicine
|July 9, 2024
Summary
Deep neuroevolution (DNE) offers a reliable uncertainty quantification (UQ) method for AI in medical imaging. This approach effectively mirrors expert assessments, enhancing trust in AI diagnostic tools.
Area of Science:
- Artificial Intelligence in Medical Imaging
- Neuroimaging and Computational Neuroscience
- Machine Learning for Clinical Decision Support
Background:
- Effective uncertainty quantification (UQ) is crucial for trustworthy artificial intelligence (AI) in high-stakes medical diagnoses.
- Current UQ methods, such as model ensembles, often suffer from impractical computational complexity and introduce invalid variability.
- There is a need for efficient and clinically viable UQ strategies that align with expert judgment.
Purpose of the Study:
- To propose and evaluate a novel UQ approach using deep neuroevolution (DNE), a data-efficient optimization strategy.
- To assess the ability of DNE-based UQ to replicate trends observed in expert-based UQ for medical imaging.
- To investigate the correlation between model-derived (epistemic) and expert-derived (aleatoric) uncertainties.
Main Methods:
- Applied DNE to generate an ensemble of 100 models for analyzing language lateralization maps from resting-state functional MRI (rs-fMRI).
- rs-fMRI data (50 maps) were split into training (30) and testing (20) sets, labeled as 'left-dominant' or 'co-dominant'.
- Model uncertainty was quantified using distribution entropies of predictions; expert uncertainties were collected for comparison.
Main Results:
- DNE achieved high accuracy in training and testing, producing a robust ensemble of models.
- Model-derived uncertainties showed consistency with expert-derived uncertainties on an in-distribution (IID) testing set.
- Both model and expert uncertainties exhibited a correlated bimodal distribution on an out-of-distribution (OOD) set, indicating varying confidence levels.
Conclusions:
- Deep neuroevolution-based UQ effectively correlates with expert assessments in medical imaging tasks.
- DNE-based UQ reliably highlights increased uncertainty in out-of-distribution medical images, crucial for clinical safety.
- This DNE approach shows promise as a reliable UQ method for radiology and other medical AI applications.
