Beyond Fidelity: Diverse Image Synthesis via Retrieval-Augmented Diffusion

Image synthesis is a key application of generative AI. It can help reduce overfitting and the high cost of collecting real-world data for downstream discriminative models. However, current methods mainly focus on making images look realistic and ignore their true goal: improving downstream model generalization and robustness. We find that existing approaches tend to produce high-fidelity synthetic images that closely resemble the original data. This limits their value for improving downstream task performance. To overcome this, we argue that diversity, not just fidelity, must guide synthetic data generation if it is to truly complement human-collected datasets. In this paper, we introduce a Retrieval-Augmented Generation framework for diverse diffusion-based image synthesis. At each generation step, we retrieve the top-K most similar samples in feature space from both real and previously generated images. We then apply an Anti-Attention mechanism that actively pushes the new image away from these retrieved samples in feature space, maximizing dissimilarity. We propose novel evaluation metrics to assess image synthesis diversity and demonstrate significant improvements over existing benchmarks. Moreover, downstream models trained with our synthetic data achieved a 1.9% absolute accuracy gain on standard benchmarks, outperforming existing synthesis techniques.