Multimodal Image Synthesis and Editing: The Generative AI Era.

Area of Science:

• Computer Vision
• Deep Learning
• Multimodal AI

Background:

• Effective interaction and fusion of multimodal information are crucial for creating and perceiving data.
• Multimodal image synthesis and editing leverage this interaction, becoming a significant research area.
• Multimodal guidance offers flexible image manipulation without explicit network training.

Purpose of the Study:

• To comprehensively review recent advancements in multimodal image synthesis and editing.
• To categorize approaches based on data modalities and model types.
• To identify current challenges and future research avenues.

Main Methods:

• Categorization of multimodal image synthesis and editing techniques by data modalities and model architectures.
• Review of different guidance modalities used in image synthesis and editing.
• Analysis of benchmark datasets, evaluation metrics, and experimental outcomes.

Main Results:

• A structured overview of the current state-of-the-art in multimodal image synthesis and editing.
• Identification of key challenges including feature alignment, high-resolution synthesis, and evaluation metrics.
• Taxonomies are formulated based on data modalities and model types.

Conclusions:

• Multimodal image synthesis and editing show great promise but face challenges in feature alignment and evaluation.
• Future research should focus on addressing these challenges to advance the field.
• The survey provides a roadmap for researchers in multimodal AI and computer vision.