3SGAN: Semi-Supervised and Multi-Task GAN for Stain Normalization and Nuclei Segmentation of Histopathological Images

Variations in staining styles-arising from differences in tissue preparation, scanners, and laboratory protocols-severely compromise the robustness of automated cell segmentation algorithms in digital pathology. Moreover, manual nucleus annotation is extremely labor-intensive, leading to a scarcity of large-scale, fully annotated datasets for supervised nucleus segmentation. This study proposes a novel framework that simultaneously mitigates staining variability and achieves high-accuracy nucleus segmentation using only minimal annotations.

We present 3SGAN, a multi-task dual-branch generative adversarial network (GAN) that jointly performs stain normalization and nucleus segmentation in a semi-supervised manner. The framework adopts a teacher-student paradigm: a lightweight teacher model (AttCycle) equipped with attention gates generates reliable pseudo-labels, while a high-capacity student model (TransCycle) leveraging a hybrid CNN-Transformer architecture further refines performance. 3SGAN was trained and evaluated on a large dataset of 1408 Whole-Slide Images (WSIs) from two medical institutions, encompassing 101 distinct staining styles, with nucleus-level annotations required for only 5% of the data.

3SGAN significantly outperformed state-of-the-art methods, achieving superior segmentation accuracy with an F1-score of 0.8140, mean IoU of 0.8201, and AJI of 0.6915. Simultaneously, it demonstrated substantial improvements in stain normalization quality, yielding a low RMSE of 0.0908, high PSNR of 21.0615, and SSIM of 0.8556 on the internal test set. External validation on independent MoNuSeg and PanNuke datasets, as well as on previously untested tumor-rich non-ROI regions from our in-house WSIs, confirmed strong generalizability with excellent stain normalization and top-tier segmentation accuracy across diverse staining protocols, tissue types, and pathological patterns.

The proposed 3SGAN framework demonstrates that high-performance nucleus segmentation and stain normalization can be achieved with minimal annotation requirements, offering a practical and scalable solution for digital pathology applications across diverse clinical settings and staining protocols.