Transformer-based Fusion of Longitudinal Multimodal Radiomic Features from Chest Radiography and CT in COVID-19

Purpose To evaluate the feasibility of a transformer structure for fusing longitudinal multimodal radiomic features from chest radiographs (CXRs) and CT images to predict outcomes and identify associated clinical events in patients with COVID-19. Materials and Methods This retrospective study analyzed de-identified longitudinal CXRs and CT images in patients with polymerase chain reaction-confirmed COVID-19. Proprietary patient data (site 1) were collected between July 2020 and May 2021, and open-access patient data (obtained before February 1, 2020) were collected from site 2. Clinical outcomes included mortality, intensive care unit admission, and ventilator use during any follow-up visit. Radiomic features were extracted from lung regions in CXRs and CT images using the Cancer Imaging Phenomics Toolkit and integrated using a transformer-based model. Patient data were partitioned into training, validation, and test sets (ratio, 65:15:20). Subgroup analyses were performed across sex, site, and modality. Model performance was assessed using area under the receiver operating characteristic curve (AUC) and weighted AUC scores, with statistical significance assessed using Student t tests. Results The study included 2274 patients (946 from site 1, 1328 from site 2; mean age, 59.84 years ± 16.84, 1171 male patients). Weighted testing AUCs for predicting outcomes were 0.86 (95% CI: 0.85, 0.86) for mortality, 0.82 (95% CI: 0.81, 0.82) for intensive care unit admission, and 0.86 (95% CI: 0.86, 0.87) for ventilator usage, outperforming models trained solely on cross-sectional data or single-modal data (P < .05). Conclusion Transformer-based fusion of longitudinal multimodal radiomic data effectively predicted clinical outcomes and events associated with COVID-19. Keywords: COVID-19, Lung, Radiomics, Multi-Head Attention, Multimodal, Longitudinal, CT, Chest Radiography Supplemental material is available for this article. © RSNA, 2026.