A hybrid ConvNeXt-ViT framework with differential evolution optimization for breast cancer classification

Breast cancer, a leading cause of mortality among women worldwide, necessitates early detection through mammography. Yet, automated classification remains challenging due to class imbalance, limited datasets, and the need for both local and global feature extraction. While convolutional neural networks (CNNs) excel in local feature extraction for mammogram classification, they struggle with long-range contextual dependencies. Conversely, transformer-based models capture global relationships effectively but require large datasets and substantial computational resources, limiting their applicability in medical imaging. To overcome these limitations, we propose DEViTNeXt, a new hybrid framework that synergistically combines ConvNeXt's convolutional efficiency with Vision Transformer (ViT) attention-based global modeling, enhanced by Differential Evolution (DE) optimization. The framework employs comprehensive preprocessing (Gaussian filtering, CLAHE enhancement) and a hybrid augmentation pipeline that integrates GAN-based synthesis of malignant cases with geometric transformations. Dual-branch feature extraction leverages ConvNeXt for hierarchical local features and ViT for global contextual relationships, with Multi-Head Attention (MHA) refinement dynamically emphasizing diagnostic regions in both branches. A DE-optimized MHA fusion layer adaptively integrates complementary. A composite loss function (Weighted Cross-Entropy + Focal Loss) addresses class imbalance while focusing on complex malignant cases. Extensive experiments on the CBIS-DDSM and MIAS datasets demonstrate DEViTNeXt's superiority, achieving 99.63% accuracy, 99.45% sensitivity, and 99.55% specificity on CBIS-DDSM under binary (Benign vs. Malignant) classification, and 98.50% accuracy on MIAS (3-class), outperforming state-of-the-art methods.