Multimodal fusion models for pulmonary embolism mortality prediction

Area of Science:

• Cardiology
• Radiology
• Artificial Intelligence in Medicine

Background:

• Pulmonary embolism (PE) is a critical cardiovascular emergency requiring accurate risk stratification for effective management.
• Current clinical practice relies on electronic health records (EHR) for context in medical imaging interpretation.
• Existing deep learning models often neglect clinical data, focusing solely on imaging pixels.

Purpose of the Study:

• To develop and compare multimodal fusion models for automatic risk stratification of PE.
• To integrate volumetric imaging data with clinical patient data for enhanced PE assessment.
• To evaluate the performance of these models in determining PE severity.

Main Methods:

• Development and comparison of multimodal fusion deep learning models.
• Utilizing both volumetric pixel data and clinical patient data.
• Employing an intermediate fusion model incorporating bilinear attention and TabNet, trained end-to-end.

Main Results:

• The best performing model achieved an area under the curve (AUC) of 0.96 for PE severity assessment.
• Multimodal data integration boosted performance by up to 14% compared to unimodal approaches.
• The model demonstrated high diagnostic accuracy with 90% sensitivity and 94% specificity.

Conclusions:

• Multimodal data fusion significantly enhances the accuracy of automatic PE severity assessment.
• The developed deep learning model shows promise for improving clinical decision-making in acute PE.
• Integrating imaging and clinical data is valuable for managing life-threatening cardiovascular emergencies like PE.