MuTriM: A multiscale deep learning model integrating longitudinal radiomics and pathomic features for predicting...

We present MuTriM, a multimodal deep learning model integrating DCE-MRI and whole-slide pathology to predict survival and radiation benefit in breast cancer.

MuTriM employs an attention-based cross-modal and cross-temporal fusion framework that concurrently integrates DCE-MRI radiomic features with pathomic features of cellular morphology on WSI. MuTriM was trained on the Fudan University Shanghai Cancer Center (FUSCC) cohort (N = 335) and externally tested on the TCGA cohort (N = 126) using matched DCE-MRI and WSI.

MuTriM was prognostic of recurrence-free survival (RFS) in a subtype-agnostic setting (HR = 5.26, 95% CI, 1.69-16.4, p = 0.004; C-index = 0.75), outperforming unimodal models based on DCE-MRI alone (C-index = 0.65) or WSI alone (C-index = 0.70). MuTriM was also prognostic among HER2 + (HR = 6.82, 95% CI, 1.08-43.2, p = 0.04) and ER+ populations (HR = 4.61, 95% CI, 1.25-17.1, p = 0.02). A high MuTriM score was associated with improved RFS (HR = 0.15, 95% CI 0.03-0.87; P = 0.03), whereas a low MuTriM score showed no significant benefit (HR = 4.06, 95% CI 0.05-31; P = 0.53) (interaction P = 0.04). Transcriptomic analyses revealed that high-risk tumors showed suppressed cytotoxic T-cell activity and enrichment of extracellular matrix remodeling, neutrophil degranulation, and neuronal signaling, consistent with immune evasion. Upregulation of NRXN1, GABRA3, KCNK3, and KLK12 was linked to worse survival, whereas SYNGR3 and CD207 were associated with favorable outcomes.

By unifying radiology, pathology, and transcriptomics, MuTriM provides an interpretable multimodal framework for breast cancer prognosis and treatment guidance.