Explainable Multimodal Deep Learning for Heart Sounds and Electrocardiogram Classification | JoVE Visualize

Area of Science:

Cardiology
Biomedical Engineering
Artificial Intelligence

Background:

Accurate classification of cardiac conditions is crucial for patient outcomes.
Multimodal analysis of phonocardiograms (PCG) and electrocardiograms (ECG) offers potential for improved diagnostic accuracy.
Deep learning models show promise in analyzing complex physiological signals.

Purpose of the Study:

To assess the performance of five distinct deep learning models for binary classification (normal/abnormal) of synchronized heart sounds and ECGs.
To evaluate the effectiveness of different model architectures, including 1D-CNN, 2D-CNN, and multimodal fusion approaches (early and late fusion).
To utilize Gradient-weighted Class Activation Mapping (Grad-CAM) to interpret model decisions and identify clinically relevant features.

Main Methods:

Developed and compared five models: ECG 1D-CNN, PCG 2D-CNN, ECG 2D-CNN, early fusion multimodal model, and late fusion multimodal model.
Applied Grad-CAM to visualize and analyze the regions of interest within ECG and PCG signals that contribute to classification decisions.
Evaluated model performance using metrics such as ROC-AUC and F1-score.

Main Results:

The early fusion multimodal model demonstrated improved performance (ROC-AUC 0.81) compared to individual signal models (ECG 1D-CNN: 0.79, PCG 2D-CNN: 0.79).
The ECG 2D-CNN achieved a higher ROC-AUC (0.82) but a lower F1-score (0.85) than the early fusion model (0.86).
Grad-CAM analysis revealed that models focus on clinically significant ECG components (QRS complex, T/P waves) and PCG sounds (S1, S2) for classification.

Conclusions:

Early fusion of synchronized ECG and PCG signals enhances binary classification performance for cardiac conditions.
While ECG 2D-CNN shows high discriminative power, multimodal fusion offers a balanced performance in terms of accuracy and robustness.
Grad-CAM confirms that deep learning models learn to identify diagnostically relevant features in cardiac signals, aligning with clinical expertise.