A layer-wise fusion network incorporating self-supervised learning for multimodal MR image synthesis.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Magnetic resonance (MR) imaging is crucial for medical diagnosis, with different modalities offering complementary information.
• Clinical limitations such as scanning time and patient conditions can lead to unavailable or low-quality MR images.
• Synthesizing missing MR modalities is essential for improving diagnostic accuracy.

Purpose of the Study:

• To develop a novel multimodal MR image synthesis network for generating missing MR images.
• To enhance the accuracy and robustness of MR image-based diagnoses.
• To address limitations of unavailable or low-quality MR data in clinical practice.

Main Methods:

• A three-stage network: feature extraction, feature fusion, and image generation.
• Utilized 2D and 3D self-supervised pretext tasks for backbone pre-training.
• Incorporated a channel attention mechanism for adaptive feature fusion and a multimodal attention feature fusion block (MAFFB).
• Employed a generative adversarial network (GAN) framework with feature-level edge information loss and pixel-wise loss.

Main Results:

• 2D and 3D self-supervised pre-training improved feature extraction and detail retention in synthetic images.
• The MAFFB effectively modeled common and unique information across modalities.
• The proposed method demonstrated high robustness in both single-modal and multimodal synthesis.
• Experimental results showed superior performance compared to state-of-the-art approaches objectively and subjectively.

Conclusions:

• The developed multimodal MR image synthesis network effectively generates high-quality, missing MR images.
• The integration of self-supervised learning and attention mechanisms enhances feature representation and fusion.
• The method offers a robust solution for improving MR imaging in clinical practice, aiding diagnosis and treatment.