Model-Based Offline Reinforcement Learning With Adversarial Data Augmentation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Model-based offline reinforcement learning (RL) aims to optimize policies using pre-collected datasets.
• Current methods struggle with static data and extrapolation errors from fixed models.
• Offline agents cannot interact with the environment for data collection.

Purpose of the Study:

• To introduce a novel approach, Model-based Offline Reinforcement learning with AdversariaL data augmentation (MORAL), to address limitations in offline RL.
• To enhance policy optimization by enriching training data through adversarial augmentation.
• To improve the robustness and applicability of model-based offline RL.

Main Methods:

• MORAL employs adversarial data augmentation, replacing fixed-horizon rollouts with alternating sampling using ensemble models.
• A dynamic adversarial process selects ensemble models against the policy to mitigate optimistic bias.
• A differential factor (DF) is integrated for regularization and error minimization during extrapolation.

Main Results:

• MORAL effectively enriches training data, enabling robust policy optimization without manual rollout horizon tuning.
• The method demonstrates adaptability across diverse offline tasks.
• Experiments on the D4RL benchmark show MORAL surpasses existing model-based offline RL techniques.

Conclusions:

• MORAL offers a significant advancement in model-based offline reinforcement learning.
• The adversarial data augmentation strategy improves policy learning and sample efficiency.
• MORAL presents a robust and broadly applicable solution for offline RL challenges.