Related Experiment Video
Updated: May 22, 2026
A Cognitive Fusion-guided Prostate Biopsy Using Multiparametric Magnetic Resonance Imaging and Transrectal Ultrasound
Published on: March 21, 2025
Preoperative identification of tumor deposits in rectal cancer using a transformer-based multimodal fusion model: a
Jiayi Xie1, Tao Jiang2, Shengming Shi1
1Department of Magnetic Resonance Imaging Diagnostic, The Second Affiliated Hospital of Harbin Medical University, No. 246, Xuefu Road, Nangang, Harbin, 150086, China.
Objective:
To develop and validate a transformer-based deep learning-radiomics model for the non-invasive preoperative discrimination of tumor deposits (TDs) in rectal cancer by integrating multi-sequence MRI features and clinical risk factors.
Methods:
This multicenter retrospective study enrolled 684 patients with pathologically confirmed rectal adenocarcinoma from three hospitals. The cohort distribution was as follows: 425 patients from Center 1 were randomly split in a 7:3 ratio into an internal training set and an internal validation set; Center 2 contributed 154 patients; and Center 3 provided 105 patients. Radiomics features (including novel topological and Hessian matrix features) and deep learning features based on DenseNet-101 were extracted from T2WI and DWI sequences, while key clinical features were screened. All features were then subjected to standardization and dimensionality reduction before being input into a self-attention-based Transformer encoder for deep fusion.Model performance was evaluated using receiver operating characteristic (ROC) analysis, decision curve analysis (DCA), calibration curves, and the net reclassification index (NRI).
Results:
The transformer-based fusion model demonstrated superior performance, achieving AUCs of 0.974, 0.742, 0.746, and 0.752 in the training, internal validation, external validation cohort 1, and external validation cohort 2, respectively. It showed optimal accuracy, stability, and the highest net clinical benefit across a wide threshold probability range. The NRI indicated a significant improvement (62.6%) over the traditional deep neural network fusion model.
Conclusions:
The MRI-based transformer multimodal fusion model enhances the capability to preoperatively identify tumor deposits in rectal cancer with high accuracy. By providing a non-invasive and reliable tool for risk stratification, this approach holds the potential to optimize individualized treatment planning and improve patient outcomes.