Attention Mask R-CNN with edge refinement algorithm for identifying circulating genetically abnormal cells

Recent studies have suggested that circulating tumor cells with abnormalities in gene copy numbers in mononuclear cell-enriched peripheral blood samples, such as circulating genetically abnormal cells (CACs), can be used as a non-invasive tool to detect patients with benign pulmonary nodules. These cells are identified through fluorescence signals counting by using 4-color fluorescence in situ hybridization (FISH) technology that exhibits high stability, sensitivity, and specificity. When FISH data are analyzed, the overlapping cells and fluorescence noise is a great challenge for identifying of CACs, thereby seriously affecting the efficiency of clinical diagnosis. To address this problem, in this study, we proposed an end-to-end FISH-based method (CACNET) for CAC identification. CACNET achieved nuclear segmentation and counted 4-color staining signals through improved Mask region-based convolutional neural network (R-CNN), followed by cell category (normal cell, deletion cell, gain cell, or CAC) according to pathological criteria. Firstly, the segmentation accuracy of overlapping nuclei was improved by adding an edge constraint head during training. Then, the interference of fluorescence noise was reduced by fusing non-local module to reconstruct the feature extraction network of Mask R-CNN. We trained and tested the proposed model on a dataset comprising 700 frames with 58,083 nuclei. The Accuracy, Sensitivity, and Specificity (overall performance metric for the algorithm) of CAC identification with CACNET were 94.06%, 92.1%, and 99.8%, respectively. Moreover, the developed method exhibited approximately identification speed of approximately 0.22 s per frames. The results showed that the proposed method outperformed the existing CAC identification methods, making it a promising approach for early screening of lung cancer.