Offline Reinforcement Learning With Behavior Value Regularization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Offline reinforcement learning (RL) enables policy learning from static datasets without real-time interaction.
• A key challenge in offline RL is extrapolation error, leading to over-optimistic Q-value estimates and performance degradation.
• Existing methods struggle to mitigate these optimistic biases effectively.

Purpose of the Study:

• To propose a novel offline actor-critic method, Offline Actor-Critic with Behavior Value Regularization (OAC-BVR).
• To address and reduce over-optimistic Q-value estimates inherent in offline RL datasets.
• To enhance the performance and reliability of policies learned in offline settings.

Main Methods:

• Introduced an offline actor-critic with behavior value regularization (OAC-BVR) framework.
• Incorporated a regularization term in the policy evaluation stage, penalizing deviations from the behavior policy's value.
• Analyzed the convergence properties of the proposed Policy Evaluation with Behavior Value Regularization (PE-BVR) component.

Main Results:

• The OAC-BVR method effectively alleviates over-optimistic Q-value estimates.
• The integration of behavior policy value regularization reduces Q-function bias.
• Experimental results on D4RL MuJoCo and Maze2d datasets show OAC-BVR outperforms state-of-the-art offline RL algorithms.

Conclusions:

• The proposed PE-BVR component is valid and contributes to improved value function estimation.
• OAC-BVR demonstrates superior performance compared to existing offline RL approaches.
• The method offers a promising direction for robust policy learning from fixed datasets.