Robust Actor-Critic With Relative Entropy Regulating Actor.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Off-policy actor-critic algorithms face challenges in accurate Q-function estimation and enhancing agent exploration.
• Existing methods struggle to balance policy stability with effective exploration strategies.

Purpose of the Study:

• To develop a novel robust actor-critic (RAC) algorithm addressing limitations in Q-function estimation and agent exploration.
• To enhance the stability and exploration capabilities of off-policy reinforcement learning agents.

Main Methods:

• A robust policy improvement mechanism (RPIM) was derived, guiding policy improvement using the local optimal policy based on the estimated Q-function.
• Relative entropy constraints were applied during policy improvement to enhance stability.
• The RPIM was integrated into the actor improvement process to create the RAC algorithm.
• Theoretical analysis confirmed the convergence of the developed RAC algorithm.

Main Results:

• The proposed RPIM enhances policy update stability by constraining relative entropy.
• Theoretical analysis indicates that policy updates incentivize increased policy entropy, boosting agent exploration.
• The RAC algorithm demonstrated superior performance compared to state-of-the-art reinforcement learning algorithms on continuous-action control tasks in the MuJoCo platform.

Conclusions:

• The novel robust actor-critic (RAC) algorithm effectively addresses challenges in Q-function estimation and agent exploration.
• RAC offers improved policy stability and exploration capabilities, leading to better performance in complex control tasks.
• The developed algorithm represents a significant advancement in off-policy reinforcement learning.