HisCMCL: cross-modal contrastive learning with hierarchical multi-scale fusion for spatial expression prediction

High costs and operational complexity limit the clinical application of spatial transcriptomics (ST). Inferring ST from pathology images is a promising alternative, the core of which lies in effectively aligning image and gene expression features. However, existing models are mostly limited to single-scale and single-slice modeling. This not only fails to connect microscopic cells with macroscopic tissues but also restricts generalization due to the inability to extract cross-sample shared features. Furthermore, the inherent representational differences between modalities further exacerbate the difficulty of feature alignment.

To address these challenges, we propose HisCMCL, a multimodal framework. The model combines multi-scale features with a cross-attention mechanism to jointly capture local morphology and global context. Additionally, it fuses spatial location information and utilizes a contrastive learning strategy to facilitate the effective alignment of image and transcriptomic features. Evaluations on four public datasets demonstrate that HisCMCL outperforms existing baseline methods in predictive performance. It exhibits good structural consistency in identifying cancer and immune markers and delineating tumor regions, offering new insights for spatial expression inference.

The code of HisCMCL is available at https://github.com/wenwenmin/HisCMCL.