An Explainable Multimodal Deep-Learning Model Using Histopathological Image and Clinical Information for Prognosis...

Accurate prognostication is crucial for the individualized management of patients with colorectal cancer (CRC), while the accuracy and interpretability of current models still need to be improved. This study aims to develop and validate an explainable multimodal model for precise survival prediction. 3381 patients with resected CRC are enrolled from three medical centers, and a multimodal deep-learning framework integrating clinical variables and histopathological images is developed. The multimodal model exhibits significantly more robust performance than unimodal models in prognosis prediction across all of the cohorts. By stratifying patients into low-risk and high-risk groups based on the model-derived risk scores, high-risk patients exhibit worse outcomes, and the stratification remains an independent prognostic factor in multivariable Cox regression. The model also identifies patients with markedly poorer survival among the stage II/III subgroup. Adjuvant chemotherapy demonstrates benefits in high-risk patients with stage III CRC but not in the low-risk group. Multi-omic analyses disclose that low-risk tumors are characterized by enhanced adaptive immune responses, whereas high-risk tumors exhibit stronger angiogenesis and higher exhaustion scores of T cells, reflecting a hypoxic and immunosuppressive microenvironment. These results support the potential of the multimodal model to effectively predict survival outcomes in resected CRC with great explainability.