Three-dimensional self-attention conditional GAN with spectral normalization for multimodal neuroimaging synthesis

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Neuroscience

Background:

• Multimodal 3D neuroimaging is crucial for understanding brain structure and function.
• Developing advanced computational models for synthesizing this data is essential for research and clinical applications.

Purpose of the Study:

• To develop and optimize a novel 3D generative adversarial network (SC-GAN) for multimodal 3D neuroimaging synthesis.
• To improve the accuracy and efficiency of generating synthetic neuroimaging data from various modalities.

Main Methods:

• Introduced a 3D conditional generative adversarial network (SC-GAN) incorporating spectral normalization and feature matching for stable training.
• Integrated a self-attention module to capture long-range dependencies within image voxels.
• Evaluated SC-GAN on the ADNI-3 dataset for predicting PET, fractional anisotropy, and mean diffusivity maps from MRI, and for super-resolution of diffusion MRI data.

Main Results:

• SC-GAN demonstrated superior performance across all evaluation metrics compared to state-of-the-art GANs, including 3D conditional GAN.
• Achieved 18%, 24%, and 29% lower prediction errors than 2D conditional GAN for fractional anisotropy, PET, and mean diffusivity tasks, respectively.
• Ablation studies confirmed adversarial learning and self-attention as key contributors to SC-GAN's enhanced performance.

Conclusions:

• Presented an efficient, end-to-end framework (SC-GAN) for multimodal 3D medical image synthesis.
• SC-GAN offers improved accuracy and robustness for generating synthetic neuroimaging data.
• The source code is publicly available to facilitate further research and development.