Multimodal Fusion of Echocardiogram Images and Electronic Medical Records for Heart Disease Screening: Retrospective...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Cardiovascular Disease Diagnostics
• Multimodal Data Integration

Background:

• Echocardiography (ECHO) is crucial for heart disease (HD) diagnosis but lacks standardized decision support.
• Current AI in imaging improves diagnosis but integrating ECHO with electronic medical records (EMRs) for screening is underexplored.

Purpose of the Study:

• To develop an explainable AI framework integrating ECHO images and EMRs for enhanced HD screening.
• To improve the interpretability of AI-driven heart disease diagnosis using multimodal data.

Main Methods:

• A retrospective study analyzed 26,936 patients with linked ECHO and EMR data.
• An explainable AI model combined image features (Inception-v3) and EMR features using extreme gradient boosting.
• Post hoc explainability was achieved using Grad-CAM for images and Shapley analysis for EMRs.

Main Results:

• The multimodal AI model achieved an AUC of 0.8147, outperforming image-only (0.7785) and EMR-only (0.7343) models.
• Explainability methods highlighted clinically relevant image regions and EMR factors like age and hypertension.
• The model showed improved sensitivity and specificity for heart disease screening.

Conclusions:

• Integrating ECHO images and EMR data via an explainable AI framework improves heart disease screening performance.
• The findings suggest clinical plausibility and potential for enhanced diagnostic workflows.
• Future research should focus on external validation and prospective clinical integration studies.