Integrating attention networks into a hybrid model for HER2 status prediction in breast cancer

Breast cancer is one of the most prevalent cancers amongst women, caused by uncontrolled cell growth in breast tissue. Human Epidermal growth factor Receptor 2 (HER2) proteins play a vital role in regulating normal breast cell development and division, and the status is crucial for determining prognosis and treatment strategies. Despite the availability of various techniques to identify the HER2 gene in tumors, the prediction accuracy of existing methods remains insufficient. This research aims to improve HER2 status prediction accuracy by proposing an Enhanced Hybrid Model with Optimized Attention Network (EHMOA-net) for histopathology image analysis. The methodology involves patch segmentation using an Encoder-Decoder-based hybrid weights alignment with Multi-Dilated U-net (EDMDU) model applied to the TCGA dataset, followed by preprocessing through enhanced Macenko stain normalization for segmented patches and images from the BCI dataset. Improved non-subsampled shearlet transform is utilized for feature extraction, and the Hybrid Enhanced Rough k-means clustering and Fuzzy C-Means (HERFCM) algorithm is employed to cluster neighboring image patches based on similar features. Finally, HER2 prediction is performed using nested graph neural networks integrated with a visual attention network. The proposed method, implemented in Python, achieves an accuracy of 97.85 %, surpassing existing techniques. These findings demonstrate the effectiveness of EHMOA-net in improving HER2 prediction accuracy and its potential utility in clinical applications.