Multiclass Classification of Renal Tumor Subtypes: Addressing Diagnostic Challenges Using a Texture-Informed Deep...

Accurate histopathological classification of renal cell carcinoma (RCC), along with its distinction from benign mimickers, is essential for precision oncology and optimal patient management. The morphological complexity and subtle differences among RCC tumors pose significant diagnostic challenges, often resulting in notable inter-observer variability. This paper presents a novel texture-informed hybrid deep learning framework that addresses these challenges by integrating a Rotation-Invariant Multi-Threshold Local Binary Pattern (RIMT-LBP) descriptor with a cascaded CNN-Transformer architecture for robust multiclass classification of renal cell neoplasms. The proposed RIMT-LBP descriptor is designed to capture multiscale tissue heterogeneity while maintaining robustness to orientation variability inherent in histopathological slides. Integrating this descriptor with the original images enriches the representation of RCC tumor features. Classification is performed using a proposed hybrid model that combines MobileNetV3Large for efficient local feature extraction and Transformer encoders for global contextual modeling. This hybrid architecture enables complementary analysis of both fine-grained cellular morphology and broader tissue architecture. The proposed model demonstrated strong whole slide image-level performance, achieving average weighted precision of 95.84%, recall of 95.36%, F1-score of 95.60%, and accuracy of 98.10%. Further analysis showed that the combined RGB+LBP approach (F1-score: 89.18%) outperformed both the RGB-only (F1-score: 84.44%) and LBP-only (F1-score: 80.31%) configurations at the patch level, confirming the complementary value of texture-informed features. Evaluation on two independent public datasets (TCGA-RCC and DHMC) confirmed the framework's consistent performance, with accuracies of 93.13% and 96.12%, respectively. These results highlight the clinical potential of integrating AI models for complementary analysis to improve RCC diagnostic accuracy.