HLIP: A pan-cancer model for histological image analysis in clinical research using TCGA

Whole slide imaging (WSI) captures high-resolution histopathological details, enabling the prediction of clinical outcomes such as tumor classification, disease progression, and patient prognosis. However, existing methods often focus on single-tumor prediction, lack pan-cancer analysis, oversimplify clinical data into categorical labels, and down-sample WSIs, leading to a loss of cellular details, and reduced accuracy. Here, we collected ∼18 K WSIs and 12 clinical features from ∼13 K patients across 32 tumor types in the TCGA database. To enhance histology-clinical associations, WSIs were divided into 512 × 512 patches, and paired with clinical features, generating 190 K histology-clinical feature pairs. We developed histology-language image pretraining (HLIP), a transformer-based model that learns embeddings from clinical paragraph descriptions and histological patches using contrastive learning. HLIP achieved F1@10 (0.886), F1@50 (0.856), and F1@100 (0.915) for zero-shot classification on external datasets, outperforming competing models by 0.6, demonstrating its strong generalization capability. Additionally, HLIP facilitates bidirectional retrieval between histology and 12 clinical features and identifies high-malignant regions in histology, highlighting its strong clinical potential.