A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains

Assess whether a single instance-segmentation model can operate robustly across multiple cytological stains, avoiding stain-specific pipelines without sacrificing accuracy.

We consolidated three expert-annotated datasets (Papanicolaou, Feulgen, AgNOR), standardised their formats and trained Mask R-CNN, Mask2Former, YOLO11n and YOLO12n under two regimes: per-stain and unified multi-stain. We reported AP₅₀ and AP₇₅ on validation/test to decouple detection tolerance from boundary precision, using identical training and evaluation protocols across models.

Feulgen specialist models yielded the highest scores in our benchmark, with YOLO11n reaching AP₇₅ 0.566 and AP₅₀ 0.643. Mask2Former led in geometric precision (AP₇₅) in complex scenarios (AgNOR and Papanicolaou), achieving the best performance on the combined dataset. Crucially, unified training matched or exceeded stain-specific models, maintaining detection (AP₅₀) while improving boundary definition (AP₇₅).

A single multi-stain model is feasible for cytology instance segmentation, preserving detection and improving boundary accuracy in diverse stains. Transformer-based Mask2Former showed the best AP₇₅ on the combined dataset, supporting a unified multi-stain, scalable path to AI-assisted cytopathology and practical screening workflows.

Can a single computer vision model handle the diversity of cytological stains found in clinical routine? This study validates a unified deep learning approach that effectively segments cells across Papanicolaou, Feulgen and AgNOR protocols. Results suggest that laboratories can deploy simplified, stain-invariant screening tools without compromising diagnostic precision, overcoming a major barrier in digital pathology interoperability.