Improving Exploration in Actor-Critic With Weakly Pessimistic Value Estimation and Optimistic Policy Optimization.

Area of Science:

• Reinforcement Learning
• Continuous Control
• Deep Learning

Background:

• Deep off-policy actor-critic methods excel in continuous control but struggle with sample efficiency.
• Poor sample efficiency limits real-world applications of current algorithms.

Purpose of the Study:

• To develop a novel actor-critic algorithm to address the sample efficiency limitations.
• To enhance exploration strategies in continuous control tasks.

Main Methods:

• Proposed Weakly Pessimistic Value Estimation and Optimistic Policy Optimization (WPVOP) algorithm.
• Integrated weakly pessimistic value estimation to encourage exploration in low-value regions.
• Employed optimistic policy optimization by sampling beneficial actions for policy learning.

Main Results:

• Theoretically analyzed the bounds of the weakly pessimistic value estimation.
• Empirically demonstrated WPVOP's ability to avoid over-optimistic value estimates.
• Achieved state-of-the-art sample efficiency and performance on MuJoCo continuous control tasks.

Conclusions:

• WPVOP effectively improves performance and sample efficiency in continuous control.
• The integration of pessimistic value estimation and optimistic policy optimization is complementary and beneficial.
• WPVOP offers a promising solution for real-world applications requiring efficient deep reinforcement learning.