MANet: a multimodal attention convolutional neural network for brain tumor classification

Classification of brain tumors is a difficult problem in medical imaging analysis. Over the past few years, various deep learning-based techniques have been employed for detecting and classifying tumors from Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). In this paper, a new model referred to as Multimodal Attention based Convolutional Neural Network (MANet) has been proposed for detection and classification of brain tumors. The proposed MANet architecture involves integrating multiple feature sets extracted from MRI images through attention techniques. These include wavelet features, edge features, and texture features. This network is divided into two modules, namely feature extraction and classification. The feature extraction module incorporates three CNN streams assisted by wavelet, edge, and texture attention, respectively. The three streams learn discriminative features from diverse feature sets. The classification module concatenates the features and makes predictions through fully connected layers. The proposed MANet models four classes of brain tumors. These classes include glioma, meningioma, pituitary, and non-tumor classes. The proposed MANet model has been tested using two publicly available datasets. These datasets include Contrast Enhanced Magnetic Resonance Imaging (CT-MRI) and Brain Tumor Magnetic Resonance Imaging (BT-MRI). Comparison has been made between MANet and several other existing methods. Experimental results show that the proposed MANet model outperforms other approaches. Comparisons have been made to many existing state-of-the-art approaches. The experimental findings suggest that the suggested MANet is more effective than other approaches. MANet model reaches the highest level of efficiency with an accuracy of 99.12%, a precision of 99.44%, a recall of 99.35%, a sensitivity of 99.1%, a specificity of 99.6%, and an F1 score of 99.5%. Furthermore, experiments show the efficacy of different attention mechanisms along with suggested multimodal fusion technique. In conclusion, the proposed MANet is very promising as a tool to classify brain tumors.