Multimodal deep learning framework for recurrence risk stratification in soft tissue sarcoma: a multicenter study

Accurate prediction of recurrence risk is essential to devise effective and personalized treatment strategies for patients with soft tissue sarcoma (STS). This study aimed to develop and validate a multimodal deep learning framework that integrates clinical features, preoperative MR images, and hematoxylin and eosin-stained whole slide images (WSIs) to predict recurrence in patients with STS. A total of 323 patients with STS were retrospectively enrolled from two hospitals, serving as development and validation sets, respectively. The ShuffleNetV2 network was utilized to develop patch-level and WSI-level signatures. A convolutional neural network fusing the channel and spatial attention mechanisms was used to develop a radiology signature. The combined model was built by integrating clinical features, radiology signature score, and WSI-level signature score with Cox regression analysis. The combined model demonstrated superior performance in the validation set, achieving a C-index of 0.857 and a time-dependent area under the curve of 0.959. Class activation maps facilitated the monitoring of suspected regions to inform recurrence decisions. The recurrence-free survival times of the low- and high-risk cohorts were statistically different (p < 0.05). The proposed multimodal framework offers satisfactory accuracy for predicting recurrence risk in patients with STS and could guide the choice of treatment modality.