Deep learning-based multimodal fusion of MRI and whole slide image for predicting neoadjuvant therapy response in...

BACKGROUND: Locally advanced head and neck squamous cell carcinoma (HNSCC) exhibits significant heterogeneity to neoadjuvant targeted therapy and chemotherapy, making personalized treatment selection challenging. This study aims to develop and validate a Transformer-based multimodal fusion model based on multimodal magnetic resonance imaging (MRI) and pathological whole slide image (WSI) to improve the prediction of neoadjuvant targeted therapy and chemotherapy response in locally advanced HNSCC. METHODS: A total of 201 patients with stage III-IV HNSCC receiving neoadjuvant targeted therapy and chemotherapy was recruited from two medical centers. For feature extraction: Macro-level imaging features were extracted from T1WI, T2WI, and contrast-enhanced T1WI (CE-T1WI) using a ResNet50-based deep learning model; Micro-level cellular features were extracted from hematoxylin and eosin (H&E)-stained WSIs via Term Frequency-Inverse Document Frequency (TF-IDF) analysis, which aggregates patch-level pathological information into slide-level representations. Based on a Transformer fusion framework with multi-head self-attention mechanisms, the multimodal fusion model dynamically weights and fuses cross-scale features across modalities. The predictive performance of each model was evaluated using the area under the curve (AUC), calibration curve, and decision curve analysis (DCA). The visualization of deep learning model was utilized to enhance interpretability. RESULTS: The multimodal fusion model outperformed single-modal models in predicting the overall response rate (ORR) of targeted therapy and chemotherapy in locally advanced HNSCC, with AUC values of 0.862 (95%CI: 0.737–0.943) and 0.825 (95%CI: 0.644–0.939) in the internal and external validation cohorts, respectively. Calibration curve and DCA further confirmed its superior clinical effectiveness. CONCLUSIONS: In summary, the MRI deep learning model and pathomics model can differentiate responders from non-responders for neoadjuvant targeted therapy and chemotherapy. The multimodal fusion model, which combines MRI and WSI features, has improved predictive performance in locally advanced HNSCC, providing an interpretable tool for personalized HNSCC treatment selection.