A Maximum Divergence Approach to Optimal Policy in Deep Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Model-free reinforcement learning (RL) often uses entropy regularization for policy learning.
• Existing methods face challenges in balancing exploration and exploitation, particularly in high-dimensional continuous spaces.

Purpose of the Study:

• To propose a novel perspective for RL by explicitly learning intrinsic information in state transitions.
• To develop a new class of Markov decision processes (MDPs) called divergence MDPs for improved policy learning.

Main Methods:

• Introduced divergence MDPs that maximize expected rewards plus a divergence term representing state transition information.
• Developed a divergence actor-critic (DivAC) algorithm using divergence policy iteration for large-scale continuous problems.

Main Results:

• The proposed DivAC algorithm demonstrated superior performance and robustness compared to existing methods in complex environments.
• The divergence term effectively captures implicit information in state transitions, leading to better stochastic policies.

Conclusions:

• Divergence MDPs offer a new framework for reinforcement learning, enhancing policy robustness and performance.
• The developed DivAC algorithm is effective for high-dimensional continuous control tasks.