Decoupled Prioritized Resampling for Offline RL.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Offline reinforcement learning (RL) is hindered by the distributional shift problem.
• Existing methods often apply uniform policy constraints, potentially harming policy learning.
• This uniform approach can negatively impact the performance of learned policies.

Purpose of the Study:

• To introduce offline decoupled prioritized resampling (ODPR) to address suboptimal policy constraints in offline RL.
• To enhance training stability through unique decoupled resampling techniques.
• To theoretically and empirically demonstrate ODPR's effectiveness in improving offline RL performance.

Main Methods:

• ODPR designs specialized priority functions to address suboptimal policy constraints.
• It employs decoupled resampling for improved training stability.
• Two implementations, ODPR-A (advantage-based) and ODPR-R (return-based), balance computation and performance.

Main Results:

• Theoretical analysis shows ODPR's priority functions improve the behavior policy distribution.
• Constraining to this improved policy is likely to yield better offline RL solutions.
• Experiments demonstrate significant performance improvements across behavior cloning (BC), TD3 BC, OnestepRL, conservative Q-learning (CQL), and implicit Q-learning (IQL) using ODPR-A and ODPR-R.
• ODPR-A shows effectiveness even without trajectory information.

Conclusions:

• ODPR is a highly compatible, plug-and-play component that significantly enhances offline RL algorithms.
• The method effectively mitigates the distributional shift problem by prioritizing actions.
• ODPR offers a promising direction for improving the robustness and performance of offline RL systems.