TexSegNet: An Attention-Guided Feedback-Driven Texture-Aware Deep Learning Model for Nuclei Segmentation and...

Accurate nuclei instance segmentation and classification play a crucial role in computational pathology, particularly for breast cancer diagnosis and characterization. However, existing methods often struggle with relatively high false positive/negative rates in detecting nuclei, inadequate nuclei texture representation leading to misclassification of nucleus types, and difficulties in properly segmenting clustered or touching nuclei in digital pathology images. In this paper, we propose TexSegNet, a hybrid encoder-decoder model that integrates multi-scale convolutions, nuclear texture extraction blocks, advanced attention mechanisms, and a feedback-driven classification branch. Trained on all tissue types included in the PanNuke dataset and subsequently fine-tuned on its breast subset, TexSegNet achieves over 4% higher accuracy in detecting and classifying nuclei on the breast test set as compared to competing models such as CellViT. Notably, TexSegNet maintains very good performance across various cell types, including underrepresented ones, with F1-scores of 89.3 ± 0.4%, 91.1 ± 0.5%, 88.9 ± 0.8%, and 84.3 ± 0.3% in detecting and classifying neoplastic, epithelial, inflammatory, and connective cell nuclei, respectively. These findings underscore TexSegNet's potential as a reliable tool for digital pathology research and as a decision-support tool to enhance diagnostic accuracy in breast histopathology.