Uncertainty estimation for deep learning-based automated analysis of 12-lead electrocardiograms

Area of Science:

• Artificial Intelligence in Medicine
• Cardiology
• Machine Learning

Background:

• Automated electrocardiogram (ECG) interpretation using deep neural networks (DNNs) shows promise but requires reliable methods for assessing result trustworthiness.
• Clinical implementation of DNNs for ECG analysis hinges on understanding and quantifying prediction uncertainty.

Purpose of the Study:

• To systematically investigate uncertainty estimation techniques for DNN-based automated ECG classification.
• To evaluate the clinical utility of uncertainty estimation through a simulation study.
• To explore the relationship between DNN uncertainty and inter-cardiologist diagnostic agreement.

Main Methods:

• Six uncertainty estimation methods (aleatoric and epistemic) were evaluated on 526,656 ECGs across three datasets.
• Methods were assessed for ranking, calibration, and robustness against out-of-distribution data.
• A clinical simulation involved applying uncertainty thresholds, and the correlation between DNN uncertainty and cardiologist agreement was analyzed.

Main Results:

• Modeling both epistemic and aleatoric uncertainty yielded the greatest benefits, with specific methods outperforming others.
• Referring uncertain predictions to physicians improved the algorithm's accuracy in the clinical simulation.
• High DNN uncertainty in ECG classification strongly correlated with reduced diagnostic agreement among cardiologists (P < 0.001).

Conclusions:

• Uncertainty estimation is essential for the reliable clinical implementation of automated DNN-based ECG classification.
• Accurate uncertainty quantification serves as a critical quality control mechanism for AI-driven ECG diagnosis.
• This work represents a significant step towards the clinical applicability of DNNs in automated ECG interpretation.