Historical Decision-Making Regularized Maximum Entropy Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• The exploration-exploitation dilemma is a key challenge in off-policy reinforcement learning (RL), hindering policy performance and sample efficiency.
• Existing RL algorithms struggle to effectively balance exploring new actions and exploiting known optimal actions.

Purpose of the Study:

• To introduce a novel algorithm, historical decision-making regularized maximum entropy (HDMRME) RL, designed to address the exploration-exploitation dilemma.
• To enhance the exploitation capabilities of RL policies within a maximum entropy framework.

Main Methods:

• Developed the HDMRME RL algorithm, integrating historical decision-making regularization into the maximum entropy RL framework.
• Conducted theoretical analysis, including convergence proofs, exploration-exploitation tradeoff analysis, Q-function disparity examination, and policy suboptimality analysis.
• Evaluated performance on continuous-action control tasks using Mujoco and OpenAI Gym platforms.

Main Results:

• HDMRME demonstrated superior sample efficiency compared to state-of-the-art RL algorithms.
• The algorithm achieved more competitive performance across various continuous-action control tasks.
• Theoretical analysis provided insights into the algorithm's convergence and tradeoff characteristics.

Conclusions:

• The HDMRME algorithm effectively balances exploration and exploitation in off-policy RL.
• HDMRME offers improved sample efficiency and performance for complex control tasks.
• This work contributes a promising new direction for advancing RL algorithm design.