Diffusion policy distillation for offline reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Offline reinforcement learning (RL) trains policies from static datasets.
• Diffusion models are effective in offline RL but suffer from slow, multi-step action sampling.
• This slowness hinders real-time control applications.

Purpose of the Study:

• To introduce a Diffusion Policy Distillation (DPD) framework.
• To accelerate decision-making in diffusion-model-based offline RL.
• To maintain policy performance while improving speed.

Main Methods:

• DPD employs a teacher-student mechanism.
• A deterministic policy (student) distills the target policy from a diffusion model (teacher).
• The distilled deterministic policy enables one-step action generation, avoiding iterative denoising.

Main Results:

• DPD achieved higher normalized scores than the original diffusion policy on D4RL Gym-MuJoCo datasets.
• The distilled policy demonstrated lower standard deviation, indicating more consistent performance.
• Decision-making speed was improved by over 10 times.

Conclusions:

• DPD effectively distills diffusion policies into faster, deterministic ones.
• The framework enhances decision-making speed without sacrificing policy performance.
• DPD is a plug-and-play solution for accelerating diffusion-model-based offline RL methods.