Mixture-of-experts and semantic-guided network for brain tumor segmentation with missing MRI modalities

Accurate brain tumor segmentation with multi-modal MRI images is crucial, but missing modalities in clinical practice often reduce accuracy. The aim of this study is to propose a mixture-of-experts and semantic-guided network to tackle the issue of missing modalities in brain tumor segmentation. We introduce a transformer-based encoder with novel mixture-of-experts blocks. In each block, four modality experts aim for modality-specific feature learning. Learnable modality embeddings are employed to alleviate the negative effect of missing modalities. We also introduce a decoder guided by semantic information, designed to pay higher attention to various tumor regions. Finally, we conduct extensive comparison experiments with other models as well as ablation experiments to validate the performance of the proposed model on the BraTS2018 dataset. The proposed model can accurately segment brain tumor sub-regions even with missing modalities. It achieves an average Dice score of 0.81 for the whole tumor, 0.66 for the tumor core, and 0.52 for the enhanced tumor across the 15 modality combinations, achieving top or near-top results in most cases, while also exhibiting a lower computational cost. Our mixture-of-experts and sematic-guided network achieves accurate and reliable brain tumor segmentation results with missing modalities, indicating its significant potential for clinical applications. Our source code is already available at https://github.com/MaggieLSY/MESG-Net .