Relative Entropy Regularized Sample-Efficient Reinforcement Learning With Continuous Actions.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Current reinforcement learning (RL) methods struggle with learning stability and sample efficiency in continuous action spaces.
• Existing actor-critic (AC) frameworks, like Deep Deterministic Policy Gradient (DDPG), face challenges in continuous control tasks.

Purpose of the Study:

• To introduce a novel RL approach, Continuous Dynamic Policy Programming (CDPP), to address stability and efficiency issues in continuous action RL.
• To enhance the actor-critic framework by integrating relative entropy regularization for improved performance.

Main Methods:

• Extended relative entropy regularization from value-based to actor-critic (AC) frameworks, specifically DDPG.
• Employed Monte Carlo estimation to handle intractable softmax operations over continuous actions.
• Utilized the Mellowmax operator and introduced a Boltzmann sampling policy for guided actor exploration.

Main Results:

• Demonstrated the positive impact of relative entropy regularization on exploration behavior and policy updates in continuous action RL.
• Achieved superior learning capability, exploration efficiency, and robustness compared to baseline methods.
• Validated the approach through benchmark and real-robot simulation tasks.

Conclusions:

• CDPP significantly enhances sample efficiency and learning stability in RL with continuous actions.
• The integration of relative entropy regularization and Boltzmann sampling offers a robust solution for complex control problems.