Characterization of tumor heterogeneity through segmentation-free representation learning

The interaction between tumors and their microenvironment is complex and heterogeneous. Recent developments in high-dimensional multiplexed imaging have revealed the spatial organization of tumor tissues at the molecular level. However, the discovery and thorough characterization of the tumor microenvironment (TME) remains challenging due to the scale and complexity of the images. Here, we propose a self-supervised representation learning framework, CANVAS, that enables discovery of novel types of TMEs. CANVAS is a vision transformer that directly takes high-dimensional multiplexed images and is trained using self-supervised masked image modeling. In contrast to traditional spatial analysis approaches which rely on cell segmentations, CANVAS is segmentation-free, utilizes pixel-level information, and retains local morphology and biomarker distribution information. This approach allows the model to distinguish subtle morphological differences, leading to precise separation and characterization of distinct TME signatures. We applied CANVAS to a lung tumor dataset and identified and validated a monocytic signature that is associated with poor prognosis.