Impact of color augmentation and tissue type in deep learning for hematoxylin and eosin image super resolution

Single image super-resolution is an important computer vision task with applications including remote sensing, medical imaging, and surveillance. Modern work on super-resolution utilizes deep learning to synthesize high resolution (HR) images from low resolution images (LR). With the increased utilization of digitized whole slide images (WSI) in pathology workflows, digital pathology has emerged as a promising domain for super-resolution. Despite extensive existing research into super-resolution, there remain challenges specific to digital pathology. Here, we investigated image augmentation techniques for hematoxylin and eosin (H&E) WSI super-resolution and model generalizability across diverse tissue types. In addition, we investigated shortcomings with common quality metrics (peak signal-to-noise ratio (PSNR), structure similarity index (SSIM)) by conducting a perceptual quality survey for super-resolved pathology images. High performing deep super-resolution models were used to generate 20X HR images from LR images (5X or 10X equivalent) for 11 different tissues and 30 human evaluators were asked to score the quality of the generated versus the ground truth 20X HR images. The scores given by a human rater and the PSNR or the SSIM were compared to investigate the correlation between model training parameters. We found that models trained on multiple tissues generalized better than those trained on a single tissue type. We also found that PSNR correlated with perceptual quality (R = 0.26) less accurately than did SSIM (R = 0.64), suggesting that the SSIM quality metric is insufficient. The methods proposed in this study can be used to virtually magnify H&E images with better perceptual quality than interpolation methods (i.e., bicubic interpolation) commonly implemented in digital pathology software. The impact of deep SISR methods is more notable when scaling to 4X is needed, such as in the case of super-resolving a low magnification WSI from 10X to 40X.