Multimodal clinical data integration for prognosis of pulmonary embolism: A comparative study

Pulmonary embolism (PE) is a life-threatening condition for which accurate risk stratification is essential for clinical decision-making. Most existing prognostic models rely on a single data modality, typically structured electronic health records (EHR), limiting their ability to capture the heterogeneous nature of the disease. Multimodal approaches combining EHR, computed tomography (CT) imaging, and free-text clinical reports provide complementary information, but the optimal fusion strategy for prognosis prediction remains unclear. In this study, we present a comprehensive comparative study of multimodal fusion strategies for PE prognosis using the INSPECT dataset, which includes structured EHR, CT images, and clinical text with mortality labels. We develop unimodal baselines and multiple bimodal and trimodal models using early, intermediate, late, and cross-fusion strategies within a unified framework. Models are evaluated for 1-, 6-, and 12-month mortality prediction. Multimodal models consistently outperform unimodal baselines across all prediction horizons, with MCC improvements up to +0.13 for 1-month mortality (0.399 vs 0.269) and +0.11 for 6-month mortality (0.479 vs 0.367). The best overall performance is achieved by late fusion models, reaching an MCC of 0.497 for 12-month prediction, compared to 0.454 for the best unimodal model. Interestingly, bimodal combinations often outperform trimodal fusion in short-term prediction, suggesting that improper integration of multiple modalities may introduce noise and reduce effectiveness. These results highlight the importance of fusion design in multimodal clinical AI and provide practical guidance for developing robust prognosis models for pulmonary embolism. Code is publicly available at https://github.com/arco-group/INSPECT-CS.