S²Net: Self-adaptive weighted fusion and self-adaptive aligned network for multi-modal MRI segmentation

Accurate segmentation of lesions is beneficial for quantitative analysis and precision medicine in multimodal magnetic resonance imaging (MRI).

Currently, multimodal MRI fusion segmentation networks still face two main issues. On one hand, simple feature concatenation fails to fully capture the complex relationships between different modalities, as it overlooks the importance of dynamically changing feature weights across modalities. On the other hand, the unlearnable nature of upsampling in segmentation networks leads to feature misalignment issues during feature aggregation with the decoder, resulting in spatial misalignments between feature maps of different levels and ultimately pixel-level classification errors in predictions.

This paper introduces the Self-adaptive weighted fusion and Self-adaptive aligned Network (S²Net), which comprises two key modules: the Self-Adaptive Weighted Fusion Module (SWFM) and the Self-Adaptive Aligned Module (SAM). S²Net can adaptively assign fusion weights based on the importance of different modalities and adaptively learn feature deformation fields to generate dynamic and flexible variability grids for feature alignment. This approach results in the generation of upsampled late-stage features with correct spatial locations and precise lesion boundaries.

This paper conducts experiments on two MRI datasets: ISLES 2022 and BraTS 2020. In the ISLES 2022 dataset, compared to the sub-optimal network MedNeXt, the proposed S²Net showed improvements of 3.52% in Dice Similarity Coefficient (DSC), 1.67% in Intersection over Union (IoU), and 4.7% in sensitivity, with a decrease of 0.33 mm in Hausdorff Distance 95 (HD95). In the BraTS 2020 dataset, compared to the sub-optimal network MedNeXt, the proposed S²Net achieved increases of 1.32% in mean DSC, 2.07% in mean IoU, and 2.17% in mean sensitivity, with a decrease of 0.10 mm in mean HD95. The code is open-sourced and available at: https://github.com/Cooper-Gu/S2Net.

Experimental results demonstrate that S²Net exhibits superior segmentation performance in multimodal MRI segmentation compared to MedNeXt, FFNet, and ACMINet.