Multimodal clinical data integration for prognosis of pulmonary embolism: A comparative study

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Cardiovascular Imaging

Background:

• Pulmonary embolism (PE) necessitates accurate risk stratification for effective clinical management.
• Current prognostic models often rely on single data sources like electronic health records (EHR), potentially missing crucial prognostic information.
• Multimodal data, including EHR, computed tomography (CT) imaging, and clinical notes, offer complementary insights but optimal integration strategies are needed.

Purpose of the Study:

• To comprehensively compare various multimodal fusion strategies for predicting pulmonary embolism (PE) patient mortality.
• To evaluate the performance of unimodal, bimodal, and trimodal models using early, intermediate, late, and cross-fusion techniques.
• To provide guidance on effective fusion design for multimodal clinical AI in PE prognosis.

Main Methods:

• Utilized the INSPECT dataset comprising structured EHR, CT images, and clinical text with mortality outcomes.
• Developed unimodal baseline models and multiple bimodal/trimodal fusion models (early, intermediate, late, cross-fusion).
• Evaluated model performance for 1-, 6-, and 12-month mortality prediction using metrics like Matthews Correlation Coefficient (MCC).

Main Results:

• Multimodal models significantly outperformed unimodal baselines across all prediction timeframes.
• Late fusion models achieved the best overall performance, with an MCC of 0.497 for 12-month mortality prediction.
• Bimodal models sometimes outperformed trimodal models in short-term predictions, indicating potential noise from suboptimal data integration.

Conclusions:

• Multimodal approaches, particularly late fusion, enhance prognostic accuracy for pulmonary embolism compared to unimodal models.
• Careful consideration of fusion strategy is critical for maximizing the benefits of multimodal data in clinical AI.
• Findings offer practical insights for developing robust PE prognosis models using diverse data sources.