Generalizable Offline Multiobjective Reinforcement Learning via Preference-Conditioned Diffuser.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Multiobjective reinforcement learning (MORL) traditionally requires costly online interaction.
• Offline MORL uses precollected datasets but struggles with generalization to new preferences.
• Existing methods lack expressiveness and perform poorly on out-of-distribution (OOD) preferences.

Purpose of the Study:

• To introduce a generalizable diffusion-based planning framework for offline MORL.
• To enhance the expressiveness and generalization capabilities of offline MORL techniques.
• To address limitations in handling diverse and OOD preferences within MORL.

Main Methods:

• Proposed diffusion-based MORL (DiffMORL) framework utilizing diffusion models.
• Implemented offline data mixup and data augmentation for improved generalization.
• Trained DiffMORL to condition on both in-distribution and OOD preferences for trajectory planning.

Main Results:

• DiffMORL achieved state-of-the-art results on the D4MORL benchmark across most tasks.
• Demonstrated superior performance in OOD generalization, outperforming baselines on 14 out of 18 metrics.
• Validated the framework's ability to plan desired trajectories and extract actions based on given preferences.

Conclusions:

• DiffMORL significantly advances offline MORL by leveraging diffusion models for enhanced planning and generalization.
• The proposed methods effectively mitigate memorization and improve feature learning through data augmentation.
• DiffMORL offers a robust solution for real-world MORL applications requiring adaptability to novel preferences.