Optimizing YOLOv11 for automated classification of breast cancer in medical images

Breast cancer diagnosis via histopathology image analysis is a complex and subjective process. While deep learning has emerged as a powerful tool for automation, achieving high accuracy across diverse cancer subtypes and magnification levels remains a significant challenge. This paper introduces a Novel-MultiScaleAttention model, an advanced architecture designed to capture discriminative features across multiple morphological scales in histopathology images. We conduct a comprehensive evaluation on two publicly available benchmark datasets: a large binary classification dataset (Breast Cancer - v1, N = 16,652 images, M_100X vs. B_100X) and the more complex 8-class subset of the BreakHis dataset (N = 4,914 images). Our proposed model is rigorously compared against state-of-the-art baselines, including YOLO11base, ResNet18, EfficientNet, and MobileNet. The results demonstrate that our model achieves superior performance, attaining a top accuracy of 0.9808 and a macro AUC of 0.9978 on the binary dataset. On the challenging 8-class dataset, it achieves a leading accuracy of 0.9363 and a macro AUC of 0.9956, outperforming other models in overall discriminative ability. Furthermore, a detailed computational analysis reveals a favorable performance-efficiency trade-off. An in-depth error analysis identifies specific misclassification patterns, aligning with known diagnostic challenges in pathology. The findings confirm that the Novel-MultiScaleAttention model provides an accurate framework for breast cancer histopathology image classification, demonstrating strong generalization capability across two distinct datasets and showing potential to serve as a valuable decision-support tool in clinical settings.