PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Diffusion models show significant advancements but degrade on imbalanced datasets.
• Performance decline is linked to disproportionate representation in image-text pairs.

Purpose of the Study:

• To propose a general fine-tuning approach to mitigate performance degradation in diffusion models caused by imbalanced datasets.
• To enhance the generation accuracy and quality of diffusion models when trained on imbalanced data.

Main Methods:

• Introduced PoGDiff, a novel fine-tuning strategy for diffusion models.
• Replaced direct KL divergence minimization with a Product of Gaussians (PoG) distribution.
• Constructed PoG by combining ground-truth targets with a predicted distribution conditioned on neighboring text embeddings.

Main Results:

• PoGDiff effectively addresses data imbalance issues in diffusion models.
• Demonstrated significant improvements in generation accuracy.
• Showcased enhanced generation quality on real-world datasets.

Conclusions:

• The proposed PoGDiff method offers a robust solution for training diffusion models on imbalanced datasets.
• PoGDiff enhances the reliability and performance of diffusion models in practical applications.
• This approach contributes to more equitable and effective generative AI development.