Toward Accurate Deep Learning-Based Prediction of Ki67, ER, PR, and HER2 Status From H&E-Stained Breast Cancer Images

Despite improvements in machine learning algorithms applied to digital pathology, only moderate accuracy, to predict molecular information from histology alone, has been achieved so far. One of the obstacles is the lack of large data sets to properly train machine learning models. We therefore built a data set of 185,538 breast cancer (BC) including hematoxylin and eosin (H&E) and associated immunohistochemistry (IHC) images of the proliferative marker Ki67, estrogen receptor (ER), progesterone receptor (PR), and the human epidermal growth factor receptor 2 (HER2). Optimal registration of H&E and IHC pairs was achieved. Ki67, ER, and PR IHC labels, to be predicted, were extracted from IHC assays using image analysis. These labels were ordinaly classified with incremental thresholds (cumulative logit models with balanced and partial proportional odds). HER2 label was determined as follows: positive if tumor IHC 3+ pattern is identified and otherwise negative. Cases with IHC equivocal score (2+) were excluded. A vision transformer (ViT)-based pipeline, trained with this data set, achieved prediction performance of 90% in terms of area under the curve (AUC) of the receiver operating characteristic (ROC) curves. ViT outperformed the weakly supervised clustering-constrained attention multiple instance learning (CLAM) which was developed to automatically identify subregions of high diagnostic value in whole slide. As a first step to "explain" artificial intelligence (AI), we evaluated the ability of both classifiers to localize these high diagnostic value subregions by inspecting their respective "attention" heat-maps. Despite high ViT AUC-ROC results, heat-maps do not obviously match areas of high diagnostic value subregions; it might however provide direction for future work to improve AI attention within whole slide images. Our proposed data set is publicly available.