MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models With Temporal Distillation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Diffusion models excel at vision generation but face computational challenges on edge devices.
• Quantization accelerates inference and reduces memory but struggles with extremely low bit-widths (2-4 bits).
• Existing quantization methods degrade performance due to outlier-unfriendly designs, suboptimal initialization, and optimization strategies.

Purpose of the Study:

• To develop an improved Mixed Precision Quantization (MPQ) framework for extremely low-bit diffusion models (MPQ-DMv2).
• To enhance quantization techniques to handle outliers and imbalanced distributions effectively.
• To optimize the training process for better convergence and temporal consistency in quantized models.

Main Methods:

• Proposed Flexible Z-Order Residual Mixed Quantization to manage salient errors from outliers using a binary residual branch.
• Introduced Object-Oriented Low-Rank Initialization, analyzing LoRA module convergence for informative initialization.
• Developed Memory-based Temporal Relation Distillation to maintain temporal consistency via an online pixel queue.

Main Results:

• MPQ-DMv2 demonstrates superior performance compared to state-of-the-art methods across various generation tasks.
• Significant improvements are observed especially under extremely low-bit width conditions (2-4 bits).
• The framework shows effectiveness across different model architectures.

Conclusions:

• MPQ-DMv2 effectively overcomes the limitations of existing quantization methods for diffusion models at extremely low bit-widths.
• The proposed techniques enable efficient and high-performance diffusion model deployment on resource-constrained edge devices.
• This work paves the way for wider adoption of diffusion models in edge AI applications.