SPACT: A clustering-driven multi-modal framework for survival prediction using genomic and histopathology data

Multi-modal data-based algorithms have gained attention in their capabilities in prediction tasks in cancer-related research. This paper introduces SPACT, a multi-modal capable of predicting cancer survival probability based on a deep-learning application on the histopathological patch features, whole-slide histopathology (WSI) in particular. What SPACT improves upon compared to the existing survival-prediction models is aiming to improve the robustness/resilience and the prediction accuracy, both overall and in subcategories. It does that by instead of relying only on the TCGA-based datasets, it benefits from the use of an external dataset collected from the Başkent Hospital. By doing cross-comparison on the different encoders' accuracy on both datasets, an optimal encoder that performs well in both the conventional TCGA and newly gathered Başkent Hospital dataset is chosen. Using a combination of multiple encoders on different models, we find that encoders that perform well on both datasets outperform the encoders that work well on a single one. Additionally, in-depth analyses on the ablation studies, risk stratification, attention maps, and on how integrated gradients may have an effect on the performance and reasoning of SPACT. SPACT matches or outperforms all the other state-of-the-art multi-modal prediction algorithms in 5 of the 7 different cancer types, particularly in the case of ovarian cancer, by a wide margin, with a c-index score of 0.77. It has the highest robustness, having the best overall performance on all 3 encoders the models have been tested with. These findings solidify SPACT's position as a contemporary and improved multi-modal based deep learning model targeting individual prediction accuracy, overall prediction accuracy, and robustness under diverse conditions. The code for SPACT is available at https://github.com/ezgiogulmus/SPACT.