Harnessing TME depicted by histological images to improve cancer prognosis through a deep learning system

Spatial transcriptomics (ST) provides insights into the tumor microenvironment (TME), which is closely associated with cancer prognosis, but ST has limited clinical availability. In this study, we provide a powerful deep learning system to augment TME information based on histological images for patients without ST data, thereby empowering precise cancer prognosis. The system provides two connections to bridge existing gaps. The first is the integrated graph and image deep learning (IGI-DL) model, which predicts ST expression based on histological images with a 0.171 increase in mean correlation across three cancer types compared with five existing methods. The second connection is the cancer prognosis prediction model, based on TME depicted by spatial gene expression. Our survival model, using graphs with predicted ST features, achieves superior accuracy with a concordance index of 0.747 and 0.725 for The Cancer Genome Atlas breast cancer and colorectal cancer cohorts, outperforming other survival models. For the external Molecular and Cellular Oncology colorectal cancer cohort, our survival model maintains a stable advantage.