Learning a Prototype Discriminator With RBF for Multimodal Image Synthesis.

Area of Science:

• Medical Imaging
• Computer Vision
• Artificial Intelligence

Background:

• Multimodal image synthesis addresses the challenge of missing data modalities.
• Existing methods using softmax classifiers struggle with intra-domain compactness, leading to suboptimal synthetic images.
• There is a need for improved domain-specific constraints in generative models.

Purpose of the Study:

• To introduce a novel prototype discriminator for generative adversarial networks (PT-GAN) to enhance multimodal image synthesis.
• To improve the optimization of generative models by incorporating domain-specific prototypes.
• To reduce the sensitivity of discriminators to pixel variations in generated images.

Main Methods:

• Developed a novel prototype discriminator for generative adversarial networks (PT-GAN).
• Integrated Radial Basis Function (RBF) networks to provide domain-specific prototypes to the discriminator.
• Introduced a reconstructive regularization term to connect the discriminator and generator, enhancing pixel detectability.

Main Results:

• PT-GAN effectively estimates missing or noisy modalities, improving synthetic image quality.
• Prototype learning emphasizes intra-domain compactness, leading to more discriminative representations.
• The method provides consistent domain-specific constraints and reasonable uncertainty estimation via RBF distance.

Conclusions:

• PT-GAN outperforms state-of-the-art techniques in multimodal image synthesis.
• The proposed approach offers a more robust and accurate method for generating images with missing modalities.
• The RBF network and reconstructive regularization contribute to enhanced generative model performance.