Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening

Background: Cervical cancer remains one of the leading causes of cancer-related morbidity among women worldwide. The Papanicolaou (Pap) smear is widely used for early detection; however, its manual interpretation is time-consuming, requires substantial expertise, and is often affected by inter-observer variability, particularly in cases with dense and overlapping cells. Methods: This study developed an optimized deep learning framework for cervical cell instance segmentation, specifically targeting the separation of overlapping cells in Pap smear images. The proposed framework was based on Mask R-CNN with a ResNet-50 backbone and Feature Pyramid Network. A public development dataset of 460 cervical smear images was used for model development and internal evaluation, while an independent 210-image dataset collected from King Abdullah University Hospital was reserved as an external held-out clinical assessment set. To improve small-cell detection and mask refinement, the Region Proposal Network was adapted using biologically informed anchor scales (16, 32, 64, 128, 256), followed by soft calibration-based post-processing. Results: The proposed framework achieved an AP50 of 89.90% and a Mask IoU of 90.44%. Small-cell performance, measured using APs under the COCO small-object convention, reached 22.80%, while APm and APl reached 68.45% and 81.42%, respectively. Clinical cell-count evaluation on the independent KAUH held-out set was performed using the mean count of five physicians as the reference standard and achieved an overall clinical detection accuracy of 93.00%. Conclusions: The optimized Mask R-CNN framework improved the detection and separation of overlapping cervical cells in Pap smear images and may serve as a supportive tool for cytopathology workflows. The results suggest that biologically informed anchor optimization and soft calibration can improve cell-level instance segmentation, particularly for small and overlapping cells. Further validation on larger multi-center datasets remains necessary before routine clinical deployment.