Patch2Space: a registration-free segmentation method for misaligned multimodal medical images

Area of Science:

• Medical image analysis
• Deep learning for medical imaging
• Computational anatomy

Background:

• Multimodal medical images offer complementary information crucial for deep learning (DL)-based segmentation.
• Accurate segmentation requires anatomical alignment via image registration, which is often challenging in clinical settings due to inconsistent fields of view (e.g., CT vs. MR).
• Image misalignment significantly degrades segmentation performance.

Purpose of the Study:

• To develop a DL-based method for segmenting misaligned multimodal images without registration.
• To learn high-quality, related features from misaligned modalities.
• To achieve segmentation accuracy comparable to methods using well-aligned images.

Main Methods:

• A unified body space (UBS) module encodes image patches from misaligned modalities and projects them into a common space, mitigating misalignment.
• A novel spatial-attention mechanism integrated into a multilevel feature fusion (MFF) module fuses features at internal, spatial, and modal levels.
• The method was validated on 1472 patients using public and in-house multimodal datasets.

Main Results:

• The proposed method achieves high accuracy in segmenting misaligned multimodal images.
• Experimental results demonstrate superior performance compared to state-of-the-art (SOTA) methods.
• Ablation studies confirmed the effectiveness of the UBS module in aligning features and the MFF module in enhancing segmentation accuracy.

Conclusions:

• The developed DL method effectively handles misaligned multimodal medical images for segmentation without requiring registration.
• The unified body space and multilevel feature fusion approach significantly improves segmentation accuracy.
• This method offers a promising solution for clinical applications where image registration is difficult.