Predicting fine-grained cell types from histology images through cross-modal learning in spatial transcriptomics

Fine-grained cellular characterization provides critical insights into biological processes, including tissue development, disease progression, and treatment responses. The spatial organization of cells and the interactions among distinct cell types play a pivotal role in shaping the tumor micro-environment, driving heterogeneity, and influencing patient prognosis. While computational pathology can uncover morphological structures from tissue images, conventional methods are often restricted to identifying coarse-grained and limited cell types. In contrast, spatial transcriptomics-based approaches hold promise for pinpointing fine-grained transcriptional cell types using histology data. However, these methods tend to overlook key molecular signatures inherent in gene expression data.

To this end, we propose a cross-modal unified representation learning framework (CUCA) for identifying fine-grained cell types from histology images. CUCA is trained on paired morphology-molecule spatial transcriptomics data, enabling it to infer fine-grained cell types solely from pathology images. Our model aims to harness the cross-modal embedding alignment paradigm to harmonize the embedding spaces of morphological and molecular modalities, bridging the gap between image patterns and molecular expression signatures. Extensive results across three datasets show that CUCA captures molecule-enhanced cross-modal representations and improves the prediction of fine-grained transcriptional cell abundances. Downstream analyses of cellular spatial architectures and intercellular co-localization reveal that CUCA provides insights into tumor biology, offering potential advancements in cancer research.

The source code of CUCA is available in Zenodo: 10.5281/zenodo.15087256.