Distributional Policy Gradient With Distributional Value Function.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Conventional reinforcement learning (RL) algorithms estimate the expected return for state-action pairs.
• Distributional RL algorithms model returns as random variables, estimating return distributions for richer environmental information.
• Existing research has extensively explored distributional RL in value-based methods, with limited application to policy-gradient methods.

Purpose of the Study:

• To bridge the research gap by proposing a novel distributional policy-gradient method.
• To enhance policy gradient estimation by incorporating distributional value functions.
• To improve exploration efficiency and escape local optima in RL tasks.

Main Methods:

• Proposed a distributional policy-gradient method (DVDPG) integrating distributional value functions into policy gradients.
• Estimated the distribution of policy gradients, rather than just the expectation.
• Introduced two policy-gradient value sampling mechanisms: distribution-probability-sampling and uniform sampling.

Main Results:

• The proposed DVDPG method enhances policy gradient stochasticity, improving exploration.
• Distribution-probability-sampling showed superior performance in sparse-reward tasks.
• Both sampling mechanisms performed similarly in dense-reward tasks, and the conventional method was shown to be a special case.
• Experimental results on OpenAI-gym tasks demonstrated the method's efficiency, outperforming baselines.

Conclusions:

• The distributional policy-gradient method (DVDPG) offers significant advantages over conventional approaches.
• The method effectively improves exploration and policy optimization in various reward settings.
• DVDPG represents a valuable advancement in reinforcement learning, particularly for policy-gradient algorithms.