Related Experiment Video
Updated: Jun 27, 2026
Author Spotlight: Advancing Alzheimer's Research – Exploring Early Detection and Multi-Omics Approaches
Published on: December 15, 2023
A multimodal approach for cardiac signals classification using deep learning with explainable AI methods
Ali Mohammad Alqudah1, Ausilah Alfraihat2
1Independent Researcher, Winnipeg, Canada.
Insights
This study introduces a novel deep learning model integrating electrocardiogram (ECG) and phonocardiogram (PCG) signals for accurate cardiovascular disease diagnosis. The multimodal approach significantly improves diagnostic performance and interpretability.
Area of Science:
- Cardiology
- Biomedical Engineering
- Artificial Intelligence
Background:
- Cardiovascular diseases (CVDs) are a major global health concern requiring precise diagnostic tools.
- Electrocardiogram (ECG) and phonocardiogram (PCG) signals offer complementary insights into cardiac electrical and mechanical functions.
- Current diagnostic methods may benefit from advanced computational approaches for improved accuracy and efficiency.
Purpose of the Study:
- To develop and validate a multimodal deep learning framework integrating ECG and PCG signals for enhanced cardiovascular disease diagnosis.
- To assess the performance of the proposed framework against single-modality and existing multimodal approaches.
- To utilize explainable AI techniques for interpreting the model's decision-making process and identifying clinically relevant features.
Main Methods:
- A dual-branch CNN-BiLSTM-SE architecture with cross-modal attention was employed to fuse ECG and PCG data.
- A comprehensive preprocessing pipeline involving wavelet denoising, adaptive filtering, and normalization was implemented.
- The model was rigorously evaluated on diverse public and custom datasets, including the MIT-BIH Arrhythmia, PTB Diagnostic ECG, and PhysioNet PCG datasets.
Main Results:
- The multimodal deep learning model achieved a high overall accuracy of 97.0% and F1-scores between 94.3% and 98.1%.
- Area Under the Curve (AUC) values exceeded 0.982 across all evaluated classes, demonstrating superior performance.
- Explainable AI methods confirmed the model's focus on clinically significant indicators like irregular R-R intervals and systolic murmurs.
Conclusions:
- The proposed multimodal deep learning framework provides a feasible, interpretable, and highly accurate decision-support system for cardiac diagnosis.
- Integration of ECG and PCG signals via advanced deep learning significantly enhances diagnostic capabilities for cardiovascular diseases.
- This approach holds promise for improving patient outcomes through more precise and timely cardiac condition identification.
Abstract:
Cardiovascular diseases remain a leading cause of mortality worldwide, necessitating accurate and timely diagnosis. Electrocardiogram (ECG) and phonocardiogram (PCG) signals provide complementary information about cardiac function, electrical and mechanical activity, respectively. In this study, we propose a multimodal deep learning framework that integrates ECG and PCG using a dual-branch CNN-BiLSTM-SE architecture with cross-modal attention. Our preprocessing pipeline includes wavelet denoising, adaptive filtering, and normalization, with parameters tuned for each dataset's noise profile. We evaluate the model on multiple datasets: MIT-BIH Arrhythmia (47 subjects), PTB Diagnostic ECG (290 subjects), PhysioNet PCG Challenge 2016 (3126 subjects), PhysioNet PCG Challenge 2022 (942 subjects), and a custom multimodal dataset (500 subjects). The model achieves an overall accuracy of 97.0%, F1-scores ranging from 94.3% to 98.1%, and AUC values above 0.982 for all classes, outperforming single-modality and existing multimodal methods. Explainable AI techniques (SHAP, Grad-CAM, Integrated Gradients) reveal that the model focuses on clinically relevant features such as irregular R-R intervals in atrial fibrillation and systolic murmurs in valvular disease. The proposed approach offers a feasible, interpretable, and accurate decision-support system for cardiac diagnosis.
