RaGAN-Seg: a relativistic adversarial enhancement and segmentation framework for nucleus segmentation in IHC images

Accurate nucleus segmentation in immunohistochemistry (IHC) images is essential for quantitative digital pathology, yet heterogeneous tissues, staining protocols, and brightfield versus fluorescence acquisition limit the transfer of models developed for routine histology. We present RaGAN-Seg, a two-stage framework in which a lightweight relativistic generative adversarial network with grouped convolutions, residual connections, and interpolative upsampling-trained with relativistic adversarial loss and R1/R2 gradient penalties-enhances nuclear contrast and boundary cues before a U-Net performs dense semantic prediction and watershed post-processing refines instance contours. In a private IHC cohort of 480 patches that span eight tissue types (four seen and four unseen) and both imaging modalities, we benchmark RaGAN-Seg against representative strong baselines, including U-Net++, Attention U-Net, SAM, Cellpose, HoVer-Net, StarDist, and DistSeg-Net, and we report computational efficiency together with ablations in the enhancement stage, adversarial objectives, and input preprocessing. On brightfield test data, RaGAN-Seg improves the best competing baseline by 11.7% in aggregated Jaccard index (AJI), 14.4% in AJI+, 2.8% in Dice, 4.0% in DQ, and 7.9% in panoptic quality; on fluorescence data, the corresponding gains are 12.9%, 12.1%, 1.2%, 13.1%, and 11.5%, respectively, and the ablations corroborate the contribution of relativistic training and enhancement. These results support a practical, generalizable approach to nuclear segmentation in IHC-centric workflows.