Spatial Omics Driven Crossmodal Pretraining Applied to Graph-based Deep Learning for Cancer Pathology Analysis.

Area of Science:

• Computational pathology
• Bioinformatics
• Machine learning in oncology

Background:

• Graph-based deep learning excels in cancer histopathology by analyzing whole slide images for outcome prediction.
• Current methods use image patch embeddings as node attributes in slide graphs.
• Spatial omics, like spatial transcriptomics, offers rich molecular data that can complement histological imaging.

Purpose of the Study:

• To explore the utility of spatial transcriptomics data combined with histological imaging.
• To develop deep learning models capable of extracting both molecular and histological information.
• To enhance graph-based learning tasks in cancer pathology by integrating multi-modal data.

Main Methods:

• Utilized a contrastive crossmodal pretraining mechanism to integrate spatial transcriptomics and histological data.
• Generated deep learning models for extracting molecular and histological features.
• Applied these models to graph-based learning tasks on histopathological slides.

Main Results:

• The proposed methods demonstrated improved performance in cancer staging, lymph node metastasis prediction, and survival prediction.
• Tissue clustering analyses also showed enhancements compared to existing methods.
• The integration of spatial omics data significantly improved graph-based deep learning models for pathology.

Conclusions:

• Leveraging spatial transcriptomics data alongside histological imaging enhances deep learning models for pathology.
• This multi-modal approach shows significant promise for improving diagnostic and prognostic accuracy in cancer workflows.
• Mining spatial omics data is a valuable strategy for advancing deep learning in digital pathology.