Toward Efficient Consistency Models via Variance-Reduced Distillation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Diffusion models excel in generative tasks but suffer from high sampling costs.
• Consistency distillation (CD) reduces sampling costs but requires extensive, computationally expensive training.
• Existing methods face substantial computational challenges in achieving comparable generative quality.

Purpose of the Study:

• Introduce variance-reduced consistency learning (vrCL) for stable and efficient consistency model training.
• Eliminate the need for repeated teacher model evaluations during distillation.
• Significantly reduce computational costs and training time for generative models.

Main Methods:

• Developed a novel distillation technique, variance-reduced consistency learning (vrCL).
• Employed a student-guided sample pair to ensure training stability.
• Removed reliance on teacher model evaluations during the training process.

Main Results:

• Achieved stable and efficient training of consistency models.
• Significantly reduced computational costs and training time.
• Attained competitive generative performance within 100,000 training iterations.

Conclusions:

• vrCL enables high training efficiency and reduced computational expense for generative models.
• The method provides a stable and effective alternative to traditional consistency distillation.
• Achieves strong generative performance with significantly less training effort.