BrainFusionNet: a deep learning and XAI model to understand local, global, and sequential features of MRI images for...

The noise of Magnetic Resonance Imaging (MRI) poses challenges for Deep Learning (DL) when tumor boundaries are obscured, tumor location and appearance are complex due to overlap between tumor and non-tumor cells, and modality identification is difficult because tumor features vanish in the later layers of the DL. Effective feature extraction from given MRI is a possible solution to overcome this challenge. Therefore, we develop BrainFusionNet that combines Convolutional Neural Networks (CNNs), Vision Transformers (ViT), and Gated Recurrent Units (GRUs) to extract spatial, contextual, and sequential features from MRI images for improved brain tumor classification. Furthermore, explainable AI such as SHAP, LIME, and Grad-CAM are integrated to visualise and highlight image regions that contribute to BrainFusionNet's decision-making process. The proposed BrainFusionNet model is evaluated on two publicly available MRI datasets. K-fold validation suggests 98% accuracy on both datasets. The model was compared with the six state-of-the-art (SOTA) CNNs and transfer learning. Among the SOTA CNNs, DenseNet121 and VGG16 achieved the highest accuracy of 96%. The novelty of BrainFusionNet is that the hybrid model effectively extracts local and global features from MRI images, even in small-scale tumor regions and small tumor sizes. The model has a balanced sequential CNN architecture to capture low-level and deeper-layer features; a customized ViT that captures local features, stabilizes gradient flow, and reduces the risk of vanishing gradients during MRI image training. The CNN and ViT outputs are fed into a GRU for final classification. Furthermore, we analyze pixel intensities to determine whether MRI image quality affects image classification. Our findings are very novel in image interpretation, as we found that the distribution of pixel intensities in MRI images affects DL performance.