Bimodal PET/MRI generative reconstruction based on VAE architectures.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Image Reconstruction

Background:

• Positron emission tomography (PET) and magnetic resonance imaging (MRI) are crucial for medical diagnostics.
• Joint reconstruction of PET/MRI data can enhance image quality and reduce acquisition times.
• Current reconstruction methods face challenges with noise and undersampling.

Purpose of the Study:

• To develop and evaluate a novel deep learning framework for synergistic PET/MRI joint reconstruction.
• To investigate the effectiveness of different variational autoencoder (VAE) architectures for multimodal data fusion.
• To compare the proposed method against traditional reconstruction techniques.

Main Methods:

• A deep learning framework integrating variational autoencoder (VAE) constraints with the alternating direction method of multipliers (ADMM) optimization.
• Exploration of three VAE architectures: joint VAE, product of experts-VAE, and multimodal JS divergence (MMJSD).
• Training and evaluation on a brain PET/MRI dataset, focusing on performance under varying acquisition times and noise levels.

Main Results:

• The synergistic approach effectively leverages information sharing between PET and MRI modalities.
• Significant improvements in peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) compared to traditional methods.
• The multimodal JS divergence (MMJSD) VAE architecture demonstrated superior performance for this methodology.

Conclusions:

• The proposed deep learning-based joint PET/MRI reconstruction method offers superior performance, particularly in challenging noisy and undersampled scenarios.
• This synergistic approach effectively compensates for missing information by integrating multimodal data.
• The findings highlight the potential of deep learning for advancing medical image reconstruction.