Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Deep generative models often re-synthesize entire images, wasting computation on unedited areas during image editing.
• Minor edits require significant computational resources due to the full re-synthesis process.

Purpose of the Study:

• To introduce Spatially Sparse Inference (SSI), a general technique to accelerate generative models.
• To reduce computational waste and latency in image editing tasks for deep generative models.

Main Methods:

• Developed Spatially Sparse Inference (SSI) to selectively compute only edited image regions.
• Implemented SSI as the Sparse Incremental Generative Engine (SIGE) for hardware acceleration.
• Cached and reused feature maps from original images for unedited regions.

Main Results:

• SIGE accelerates DDPM by 3.0×-4.6×, Stable Diffusion by 7.2×, and GauGAN by 5.2×-5.6× on various GPUs.
• Significant speedups achieved with minimal edit areas (approx. 1%).
• Enhanced SIGE to support attention layers and Apple M1 Pro GPUs.

Conclusions:

• Spatially Sparse Inference (SSI) and SIGE offer a computationally efficient approach to image editing with deep generative models.
• The method effectively reduces latency without compromising output quality for various models like diffusion and GANs.