Transferable deep learning with coati optimization algorithm based mitotic nuclei segmentation and classification...

Image processing and pattern recognition methods have recently been extensively implemented in histopathological images (HIs). These computer-aided techniques are aimed at detecting the attentive biological markers for assisting the final cancer grading. Mitotic count (MC) is a significant cancer detection and grading parameter. Conventionally, a pathologist examines the biopsy image physically by employing higher-power microscopy. The MC cells have been marked physically at every analysis, and total MC must be utilized as a major aspect for the cancer ranking and considered as the initiative of cancers. Numerous pattern recognition algorithms for cell-sized objects in HIs depend upon segmentation to assess features. The correct description of the segmentation has been difficult, and feature outcomes can be highly complex to the segmentation. The MC cells are an essential element in many cancer grading methods. Extraction of the MC cell from the HI is a highly challenging assignment. This manuscript proposes the Coati Optimization Algorithm with Deep Learning-Driven Mitotic Nuclei Segmentation and Classification (COADL-MNSC) technique. The major aim of the COADL-MNSC technique is to utilize the DL model to segment and classify the mitotic nuclei (MN). In the preliminary stage, the COADL-MNSC approach implements median filtering (MF) for pre-processing. Besides, the COADL-MNSC approach utilizes the Hybrid Attention Fusion U-Net (HAU-UNet) model to segment the MN. Moreover, the capsule network (CapsNet) model is employed for the feature extraction method, and its hyperparameters are adjusted by utilizing the COA model. At last, the classification procedure is performed using the bidirectional long short-term memory (BiLSTM) model. Extensive simulations are performed under the MN image dataset to exhibit the excellent performance of the COADL-MNSC methodology. The experimental validation of the COADL-MNSC methodology portrayed a superior accuracy value of 98.89% over existing techniques under diverse measures.