Optimized segmentation of overlapping cervical cells based on Mask2Former and denoising

Cell instances segmentation in cervical cytology images have crucial importance for cancer screening and automatic biomedical image analysis. However, that the highly overlapping cells cause ambiguous boundaries makes accurate cell segmentation still a challenging task. This paper proposes a novel transformer-based neural network to segment overlapping cervical cells, by using the technique of denoising with Mask2Former. The model optimization includes two aspects. One is that class embeddings of ground truth categories are used as extra content queries to transformer decoder. The other is that noised ground truth masks are fed into each layer of transformer decoder and then the model is trained to reconstruct the original masks. Compared with the best result on ISBI2014 dataset, the proposed model obtains performance improvement by [Formula: see text] on DSC, [Formula: see text] on TPRp and [Formula: see text] on FNRo, respectively. Especially, the masks predicted by our model are more precise than other methods and our previous models. Those results show that the proposed method could be effective in segmenting overlapping cells and help pathologists detect cell lesions.