Imprint cytology-based breast malignancy screening: an efficient nuclei segmentation technique

Imprint cytology (IC) refers to one of the most reliable, rapid and affordable techniques for breast malignancy screening; where shape variation of H&E stained nucleus is examined by the pathologists. This work aims at developing an automated and efficient segmentation algorithm by integrating Lagrange's interpolation and superpixels in order to delineate overlapped nuclei of breast cells (normal and malignant). Subsequently, a computer assisted IC tool has been designed for breast cancer (BC) screening. The proposed methodology consists of mainly three subsections: gamma correction for preprocessing, single nuclei segmentation and segmentation of overlapping nuclei. Single nuclei segmentation combines histogram-based thresholding and morphological operations; where segmentation of overlapping nuclei includes concave point detection, Lagrange's interpolation for overlapping arc area detection and the fine segmentation of overlapped arc area by superpixels. Total 16 significant features (p < 0.05) quantifying shape and texture of nucleus were extracted, and random forest (RF) classifier was skilled for automated screening. The proposed methodology has been tested on 120 IC images (approximately 12 000 nuclei); where 98% segmentation accuracy and 99% classification accuracy were achieved. Besides, performance evaluation was studied by using Jaccard's index (= 94%), correlation coefficient (= 95%), Dice similarity coefficient (= 97%) and Hausdorff distance (= 43%). The proposed approach could offer benefit to the pathologists for confirmatory BC screening with improved accuracy and could potentially lead to a better shape understanding of malignant nuclei.