Optimized segmentation of overlapping cervical cells based on Mask2Former and denoising

Area of Science:

• Biomedical image analysis
• Computational pathology
• Artificial intelligence in healthcare

Background:

• Accurate cell segmentation in cervical cytology is vital for cancer screening.
• Overlapping cells and ambiguous boundaries present significant challenges for automated analysis.
• Existing methods struggle with precise segmentation of densely packed cells.

Purpose of the Study:

• To develop a novel transformer-based neural network for accurate segmentation of overlapping cervical cells.
• To improve the precision of cell instance segmentation in challenging cytological images.
• To enhance automated analysis for more effective cancer screening.

Main Methods:

• Proposed a novel transformer-based neural network incorporating denoising techniques with Mask2Former.
• Integrated class embeddings of ground truth categories as extra content queries in the transformer decoder.
• Employed a denoising strategy where noised ground truth masks are fed into the transformer decoder for reconstruction.

Main Results:

• Achieved performance improvements of [Formula: see text] on DSC, [Formula: see text] on TPRp, and [Formula: see text] on FNRo compared to the state-of-the-art on the ISBI2014 dataset.
• Demonstrated more precise mask predictions than existing methods and previous models.
• Validated the effectiveness of the proposed method in segmenting overlapping cells.

Conclusions:

• The novel transformer-based approach effectively segments overlapping cervical cells, addressing a key challenge in cytopathology.
• The method shows significant potential to aid pathologists in detecting cellular lesions more accurately.
• This advancement contributes to improved automated biomedical image analysis for cancer screening.