Off-OAB: Off-Policy Policy Gradient Method With Optimal Action-Dependent Baseline.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Policy-based methods are successful in reinforcement learning (RL).
• Off-policy policy gradient (OPPG) methods leverage off-policy data but suffer from high variance.
• High variance leads to poor sample efficiency in training.

Purpose of the Study:

• To propose a novel off-policy policy gradient method to mitigate variance.
• To introduce an optimal action-dependent baseline (Off-OAB) for unbiased and low-variance OPPG estimation.
• To enhance computational efficiency through an approximated optimal baseline.

Main Methods:

• Developed an optimal action-dependent baseline (Off-OAB) for OPPG.
• Theoretically minimized variance while maintaining unbiasedness.
• Designed an approximated version of the optimal baseline for practical efficiency.

Main Results:

• The Off-OAB method demonstrably reduces OPPG estimator variance.
• Evaluated on six OpenAI Gym and MuJoCo tasks.
• Outperformed state-of-the-art methods on most tasks.

Conclusions:

• The proposed Off-OAB method effectively reduces variance in off-policy policy gradient estimation.
• Off-OAB enhances sample efficiency and performance in challenging RL tasks.
• The approximated baseline ensures practical computational efficiency.