A Vision Transformer- and Radiomics-based Model for Predicting Liver Metastasis-Free Survival in Patients with Rectal...

Purpose To develop and validate a multimodal random survival forest (RSF) model integrating three-dimensional vision transformer (ViT-3D) architecture, radiomic features, and clinical variables derived from rectal MR images to predict liver metastasis-free survival (LMFS) in patients with rectal cancer (RC). Materials and Methods This retrospective study included patients with pathologically confirmed RC treated at three institutions from January 2019 to December 2022. A multimodal RSF model (RSF-combined) incorporating ViT-3D-derived deep learning features, radiomic features, and clinical variables was developed. Comparator RSF, extreme gradient boosting, and Cox proportional hazards models using clinical and MRI-derived data were also constructed. Model performance was evaluated using time-dependent receiver operating characteristic analysis. Model interpretability was assessed using Shapley additive explanations analysis to quantify feature contributions to individual risk predictions. Results A total of 548 patients with RC (mean age ± SD, 61.68 years ± 9.93; 356 males) were included (404 for training, 144 for testing). The RSF-combined model outperformed Cox and extreme gradient boosting models in LMFS prediction across all time points, achieving area under the receiver operating characteristic curve values of 0.72, 0.78, and 0.73 at 1, 3, and 5 years, respectively, in the external test set. Shapley additive explanations summary plots demonstrated how individual feature values dynamically influenced predicted LMFS risk. Risk stratification based on the optimal cutoff and the median RSF-combined risk score yielded significant separation of LMFS curves (log-rank test, hazard ratio: 2.09 [95% CI: 1.20, 3.27], P = .002, and hazard ratio: 1.92 [95% CI: 1.13, 3.25], P = .02, respectively). Conclusion A multimodal RSF-combined model integrating deep learning, radiomic, and clinical features enabled accurate, individualized prediction of postoperative LMFS in patients with RC. Keywords: Deep Learning, Liver Metastasis, Liver Metastasis-Free Survival, MRI, Radiomics, Rectal Cancer Supplemental material is available for this article. © RSNA, 2026.