Beyond Fidelity: Diverse Image Synthesis via Retrieval-Augmented Diffusion

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Generative AI is crucial for image synthesis, aiding in reducing overfitting and data collection costs for discriminative models.
• Current image synthesis methods prioritize realism, potentially limiting their effectiveness in improving downstream model generalization and robustness.
• High-fidelity synthetic images that closely resemble original data may not sufficiently enhance downstream task performance.

Purpose of the Study:

• To introduce a novel framework for diverse diffusion-based image synthesis that prioritizes data diversity over mere fidelity.
• To enhance the generalization and robustness of downstream discriminative models using synthetic data.
• To develop new evaluation metrics for assessing image synthesis diversity.

Main Methods:

• A Retrieval-Augmented Generation (RAG) framework is proposed for diffusion-based image synthesis.
• At each generation step, top-K similar samples are retrieved from real and generated images in feature space.
• An Anti-Attention mechanism is employed to maximize dissimilarity by pushing new images away from retrieved samples.

Main Results:

• The proposed method significantly improves image synthesis diversity compared to existing benchmarks.
• Novel evaluation metrics demonstrate enhanced diversity assessment.
• Downstream models trained with the generated diverse synthetic data achieved a 1.9% absolute accuracy gain on standard benchmarks.

Conclusions:

• Diversity, not just fidelity, is essential for synthetic data to effectively complement real-world datasets.
• The proposed Retrieval-Augmented Generation framework with Anti-Attention offers a promising approach for diverse image synthesis.
• The method demonstrably improves downstream model performance, outperforming existing synthesis techniques.