Aligning Text-to-Image Diffusion Models With Constrained Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Reward finetuning aligns diffusion models with user preferences.
• Current methods face reward overoptimization, degrading overall performance.
• Overoptimization stems from granularity mismatch, plasticity loss, and narrow objective focus.

Purpose of the Study:

• Introduce Constrained Diffusion Policy Optimization (CDPO) to mitigate reward overoptimization in diffusion models.
• Address the limitations of existing reward finetuning techniques.
• Enhance model alignment and generalization capabilities.

Main Methods:

• Developed a novel reinforcement learning framework, CDPO.
• Implemented temporal policy optimization for step-specific rewards.
• Incorporated a neuron reset strategy to maintain model plasticity.
• Integrated constrained reinforcement learning with auxiliary reward objectives.

Main Results:

• CDPO effectively reduces reward overoptimization in diffusion models.
• Temporal rewards mitigate overfitting to sparse, final-step feedback.
• Neuron resets prevent overoptimization caused by plasticity loss.
• Constrained optimization ensures balanced performance across multiple criteria.

Conclusions:

• CDPO offers a robust solution to reward overoptimization in diffusion model finetuning.
• The framework enhances model alignment without sacrificing performance or generalization.
• CDPO advances the field of reinforcement learning for generative models.