Hierarchical in-out fusion for incomplete multimodal brain tumor segmentation

Fusing multimodal data play a crucial role in accurate brain tumor segmentation network and clinical diagnosis, especially in scenarios with incomplete multimodal data. Existing multimodal fusion models usually perform intra-modal fusion at both shallow and deep layers relying predominantly on traditional attention fusion. Rather, using the same fusion strategy at different layers leads to critical issues, feature redundancy in shallow layers due to repetitive weighting of semantically similar low-level features, and progressive texture detail degradation in deeper layers caused by the inherent feature of deep neural networks. Additionally, the absence of intra-modal fusion results in the loss of unique critical information. To better enhance the representation of latent correlation features from every unique critical features, this paper proposes a Hierarchical In-Out Fusion method, the Out-Fusion block performs inter-modal fusion at both shallow and deep layers respectively, in the shallow layers, the SAOut-Fusion block with self-attention extracts texture information; the deepest layer of the network, the DDOut-Fusion block which integrates spatial and frequency domain features, compensates for the loss of texture detail by enhancing the detail of the high frequency component. which utilizes a gating mechanism to effectively combine the tumor's positional structural information and texture details. At the same time, the In-Fusion block is designed for intra-modal fusion, using multiple stacked Transformer-CNN blocks to hierarchical access modality-specific critical signatures. Experimental results on the BraTS2018 and the BraTS2020 datasets validate the superiority of this method, demonstrating improved network robustness and maintaining effectiveness even when certain modalities are missing. Our code is available https://github.com/liufangcoca-515/InOutFusion-main .