A new imbalance-aware loss function to be used in a deep neural network for colorectal polyp segmentation

Colorectal cancers may occur in colon region of human body because of late detection of polyps. Therefore, colonoscopists often use colonoscopy device to view the entire colon in their routine practice to remove polyps by excisional biopsy. The aim of this study is to develop a new imbalance-aware loss function, i.e., omni-comprehensive loss, to be used in deep neural networks to overcome both imbalanced dataset and the vanishing gradient problem in identifying the related regions of a polyp. Another reason of developing a new loss function is to be able to produce a more comprehensive one that has evaluation capabilities of region-based, shape-aware, and pixel-wise distribution loss approaches at once. To measure the performance of the new loss function, two scenarios have been conducted. First, an 18-layer residual network as backbone with UNet as the decoder is implemented. Second, a 34-layer residual network as the encoder and a UNet as the decoder is designed. For both scenarios, the results of using popular imbalance-aware losses are compared with those of using our proposed new loss function. During training and 5-fold cross validation steps, multiple publicly available datasets are used. In addition to original data in these datasets, their augmented versions are also created by flipping, scaling, rotating and contrast-limited adaptive histogram equalization operations. As a result, our proposed new custom loss function produced the best performance metrics compared with the popular loss functions.