TexTNet: a texture-guided hierarchical architecture for prostate cancer Gleason segmentation in whole slide images

Prostate cancer is one of the most common malignancies among men, and accurate segmentation of Gleason grades is crucial for effective diagnosis and treatment planning. Current deep learning methods often struggle with fusing complementary features from CNNs and Transformers, primarily relying on naive feature fusion techniques such as concatenation or summation. These methods fail to leverage the complementary strengths of texture and global context, limiting segmentation performance. To address this challenge, we propose TexTNet, a novel texture-guided hierarchical architecture that integrates Local Binary Pattern (LBP) features into the query projection of the Swin Transformer, creating a Texture-Guided Attention mechanism. This approach directs the Transformer's attention to texture-rich regions such as glandular borders and nuclear clusters, improving segmentation accuracy. We further enhance feature fusion with the Dual Path Squeeze-Fusion-Excitation (DP-SFE) module, which mitigates boundary blurring, and introduce Asymmetric Convolution Blocks (ACB) for efficient feature extraction. Additionally, the Cross-Directional Feature Aggregator (CDFA) ensures precise positional reconstruction and noise suppression during decoder fusion. We evaluate TexTNet on the SICAPv2, PANDA, ProstateX, and HuBMAP datasets, demonstrating superior performance over state-of-the-art methods in prostate cancer segmentation. TexTNet shows promise as a clinical decision-support tool, demonstrating robust performance on complex, real-world histopathological images and the potential to improve diagnostic reliability for prostate cancer grading.