Hybrid CNN-transformer model with gated feature fusion and GAN-based augmentation for MRI image classification
Related Experiment Videos
Sedat Orenc1, Cebrail Turkeri2,3, Yusuf Alaca4
1Department of Electrical and Electronics Engineering, Batman University, Batman, Turkey.
Objective:
Brain tumor classification based on magnetic resonance imaging (MRI) plays a critical role in clinical diagnosis, yet it remains challenging due to high intra-class variability, inter-class similarity, and limited annotated data. To address these challenges, this study proposes a gated hybrid convolutional neural network (CNN)-transformer architecture (GHCT-Net) that effectively integrates local and global feature representations for robust brain tumor classification in data-limited medical imaging scenarios.
Methods:
The proposed framework combines a ConvNeXt-based convolutional branch for fine-grained spatial feature extraction with a Swin Transformer branch for modeling long-range contextual dependencies. A gate-based fusion mechanism is introduced to adaptively balance the contribution of each feature stream. Additionally, a conditional deep convolutional generative adversarial network (cDCGAN) is employed to generate class-consistent synthetic MRI samples to mitigate data scarcity. The model was evaluated on a public dataset of 7023 MRI images across four classes (glioma, meningioma, pituitary tumor, and no tumor) under three training scenarios: original, traditionally augmented, and GAN-augmented datasets.
Results:
The results demonstrate that the GHCT-Net achieves superior performance compared to state-of-the-art CNN and Transformer architectures, reaching a classification accuracy of 98.39% on the GAN-augmented dataset. Quantitative analysis, along with confusion matrices, t‑distributed stochastic neighbor embedding (t-SNE) visualizations, and gradient-weighted class activation mapping (Grad-CAM) interpretability studies, confirms that the model successfully learns discriminative and clinically relevant features.
Conclusion:
Combining a gated hybrid CNN-transformer architecture with GAN-based data augmentation provides a highly effective and robust solution for brain tumor classification, particularly in scenarios constrained by limited medical imaging data.