Rectifying GAN Gradient Supervision Via Score Functions

Generative adversarial networks (GANs) and diffusion probabilistic models (DPMs) have emerged as the most prominent paradigms in generative modeling, and recent works suggest that combining GANs and DPMs could further enhance generative models. While the adversarial training of GANs has been applied to knowledge distillation of DPMs in the literature, the potential of leveraging DPMs to improve GAN training remains under-explored. In this work, we demonstrate the theoretical connections between GANs and DPMs, bridging the two most important paradigms of generative models for the first time. Specifically, we show that the generator supervision from the discriminators is equivalent to the score functions from DPMs. As a result, we propose a universal solution to stabilize GAN training, GraSF, which supervises the generator through backward propagation with gradients computed by pre-trained DPMs in a closed-form manner, with almost no increase in computational cost. Regarding the experimental evidence, we first design an illustrative example to show that our GraSF helps the generator achieve global optimality even if the discriminator suffers from non-optimality. As for real-world datasets, we manage to reach the 1.96 FID on ImageNet 64x64 dataset, achieving the best performance among all GAN-based methods with no pre-trained ImageNet prior, even surpassing some leading DPMs. This significant narrowing of the performance gap between GANs and DPMs confirms the practical value of our theoretical results and the efficacy of our method. Source code has been uploaded to IEEE Xplore Code Ocean.