Multimodal fusion of ultrasound images using HXM net for breast cancer diagnosis

Breast cancer is a major global health issue in women, where diagnosis at an early stage is decisive for enhancing the effectiveness of treatment and survival. Despite the advances in imaging using medical technologies, maintaining uniformly good diagnostic accuracy is challenging due to the difficulty of lesion characterization and limitations of single-modality imaging. This work presents HXM-Net, a deep learning model specifically tailored to improve breast cancer detection through the synergistic benefit of multi-modal ultrasound imaging. HXM-Net combines Convolutional Neural Networks (CNNs) for spatial feature extraction and Transformer-based fusion for optimal concatenation of information from the B-mode and Doppler ultrasound images. The dual modality-based method captures morphological and vascular features of breast lesions and produces a more informative and discriminative feature representation. The model embeds multi-scale feature learning and data augmentation to guarantee generalizability to different lesion types and patient populations. In the tests conducted on a class-balanced breast ultrasound database, the HXM-Net achieved accuracies of 94.20%, sensitivity (recall), 92.80%, specificity, 95.70%, an F1 score of 91.00%, and AUC-ROC of 0.97, thereby establishing the superiority of HXM-Net over conventional models like ResNet-50 and U-Net, especially in distinguishing benign tumours from malignant ones. The strong diagnostic ability and versatility of the model make it a good contender for integration with clinical decision support systems that can facilitate more assured diagnostics and enhanced patient care.