Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening

Area of Science:

• Medical Imaging
• Computational Pathology
• Artificial Intelligence in Healthcare

Background:

• Cervical cancer is a significant global health concern.
• Manual interpretation of Papanicolaou (Pap) smears is challenging due to cell density and overlapping, leading to variability.
• Automated analysis of Pap smears is needed to improve efficiency and accuracy.

Purpose of the Study:

• To develop an optimized deep learning framework for cervical cell instance segmentation.
• To specifically address the challenge of separating overlapping cells in Pap smear images.
• To enhance the detection and segmentation of small cervical cells.

Main Methods:

• Utilized a Mask R-CNN framework with a ResNet-50 backbone and Feature Pyramid Network.
• Adapted the Region Proposal Network with biologically informed anchor scales for improved small-cell detection.
• Implemented soft calibration-based post-processing for mask refinement.
• Evaluated the model on public and independent clinical datasets.

Main Results:

• Achieved high performance metrics: AP50 of 89.90% and Mask IoU of 90.44%.
• Demonstrated improved performance for small cells (AP 22.80%) and medium/large cells (APm 68.45%, APl 81.42%).
• Clinical evaluation on an independent dataset showed an overall detection accuracy of 93.00% compared to expert physicians.

Conclusions:

• The optimized Mask R-CNN framework effectively improves the detection and separation of overlapping cervical cells.
• Biologically informed anchor optimization and soft calibration enhance cell-level instance segmentation, especially for challenging cases.
• The framework shows potential as a supportive tool in cytopathology workflows, warranting further multi-center validation.