Unified Multi-Modal Image Synthesis for Missing Modality Imputation

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Multi-modal medical images offer complementary diagnostic information but are often incomplete due to practical limitations.
• Incomplete imaging data restricts the full utilization of multi-modal datasets in clinical settings.
• Existing methods struggle with synthesizing missing modalities from arbitrary combinations of available ones.

Purpose of the Study:

• To develop a unified method for synthesizing missing medical image modalities from any available subset.
• To enable robust multi-modal medical image completion using a single generative model.
• To enhance the clinical utility of multi-modal imaging by addressing data incompleteness.

Main Methods:

• A novel generative adversarial network (GAN) architecture is proposed for multi-modal image synthesis.
• A Commonality- and Discrepancy-Sensitive Encoder is designed to leverage both shared and unique information across modalities.
• A Dynamic Feature Unification Module integrates features from a variable number of input modalities, handling missing data robustly.

Main Results:

• The proposed method successfully synthesizes missing modalities from various combinations of available inputs using a single model.
• The Commonality- and Discrepancy-Sensitive Encoder ensures anatomical consistency and realistic image details.
• The Dynamic Feature Unification Module effectively integrates information, demonstrating robustness to random missing modalities.
• Experiments on two public multi-modal MRI datasets show superior performance compared to existing methods across diverse synthesis tasks.

Conclusions:

• The developed unified multi-modal image synthesis method effectively imputes missing modalities.
• The novel encoder and feature unification module enable robust and accurate synthesis from incomplete data.
• This approach significantly advances the potential of using incomplete multi-modal medical images for clinical applications.