A hybrid quantum-classical framework for MRI-based deep brain tumor segmentation and classification

The characterization of brain tumors from magnetic resonance imaging (MRI) is crucial for diagnosis, treatment planning, and prognosis. But the analysis of brain tumors based on MRI is still difficult because of the significant intra-tumoral heterogeneity, ambiguous boundaries of lesions, high dimensionality of imaging features, and complicated nonlinear relationships between anatomical structures. While transformer-based deep learning (DL) models have greatly advanced automated tumor analysis, classical feature representations often fail to effectively capture complex feature interactions in pathological images, especially when distinguishing between subtle pathological variations and normal tissue needs to be done efficiently. Recent quantum machine learning (QML) developments indicate that quantum feature encoding can map complex data to exponentially larger Hilbert spaces, leading to a more expressive representation of nonlinear patterns and feature correlations that are challenging to model with conventional architectures. Encouraged by this potential, this study proposes a hybrid quantum-classical framework, named QFormer-Brain, which combines transformer-based segmentation with quantum representation learning for automated brain tumor segmentation and classification. To segment the tumor regions, a Shifted Window UNet Transformers (Swin-UNETR) architecture is first used to capture both local anatomical details and global contextual information through hierarchical self-attention. Then, the deep semantic features of the segmented lesions are mapped to the quantum feature mapping module with angle encoding and variational quantum circuits (VQCs), which can obtain richer nonlinear feature representation by quantum superposition and entanglement mechanisms. The obtained quantum embeddings are then combined with features obtained from the transformer and classified by a Quantum Transformer Classifier. The experiments were performed on the publicly available BraTS 2021 data set using an 8-qubit variational quantum circuit, with 1024 measurement shots and readout-error mitigation, via the IBM Qiskit Aer simulator. The proposed framework achieved a Dice Similarity Coefficient (DSC) of 97.1%, Intersection-over-Union (IoU) of 95.0%, classification accuracy of 98.5%, sensitivity of 98.1%, specificity of 99.0%, and F1-score of 98.3%, outperforming U-Net, nnU-Net, ResNet50, Vision Transformer (ViT), and conventional hybrid Convolutional Neural Network (CNN)-based models. These results show that quantum-inspired representation learning can be efficiently integrated into transformer-based models to boost the discriminative feature extraction from complex MRI data.