The Diffusion Duality.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Uniform-state discrete diffusion models offer fast text generation but are often outperformed by autoregressive and masked diffusion models.
• Existing discrete diffusion models face challenges in matching the performance of continuous Gaussian diffusion models.

Purpose of the Study:

• To narrow the performance gap between uniform-state discrete diffusion models and autoregressive models.
• To enhance both the training and sampling efficiency of discrete diffusion models.

Main Methods:

• Leveraging the insight that uniform-state diffusion processes emerge from underlying Gaussian diffusion.
• Introducing a curriculum learning strategy guided by Gaussian processes to reduce variance and accelerate training.
• Adapting consistency distillation from continuous to discrete settings via Discrete Consistency Distillation for accelerated sampling.

Main Results:

• Curriculum learning doubled training speed and improved zero-shot perplexity, surpassing autoregressive models on 3 of 7 benchmarks.
• Discrete Consistency Distillation enabled few-step generation in diffusion language models, accelerating sampling by two orders of magnitude.
• The Duo method effectively transfers techniques from Gaussian diffusion to discrete settings, enhancing model performance.

Conclusions:

• The Duo method significantly improves uniform-state discrete diffusion models, making them competitive with autoregressive models.
• Efficient training and accelerated few-step sampling are key advancements for diffusion language models.
• The proposed techniques offer a promising direction for developing faster and more effective text generation models.