Universal encoding of pan-cancer histology by deep texture representations

Cancer histological images contain rich biological and clinical information, but quantitative representation can be problematic and has prevented the direct comparison and accumulation of large-scale datasets. Here, we show successful universal encoding of cancer histology by deep texture representations (DTRs) produced by a bilinear convolutional neural network. DTR-based, unsupervised histological profiling, which captures the morphological diversity, is applied to cancer biopsies and reveals relationships between histologic characteristics and the response to immune checkpoint inhibitors (ICIs). Content-based image retrieval based on DTRs enables the quick retrieval of histologically similar images using The Cancer Genome Atlas (TCGA) dataset. Furthermore, via comprehensive comparisons with driver and clinically actionable gene mutations, we successfully predict 309 combinations of genomic features and cancer types from hematoxylin-and-eosin-stained images. With its mounting capabilities on accessible devices, such as smartphones, universal encoding for cancer histology has a strong impact on global equalization for cancer diagnosis and therapies.