Discretizing Continuous Action Space With Unimodal Probability Distributions for On-Policy Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Discretizing continuous action spaces in on-policy reinforcement learning (RL) simplifies optimization but can lead to high variance due to ignored action ordering.
• The explosion of discrete actions without considering their inherent order can negatively impact policy gradient (PG) estimator performance.

Purpose of the Study:

• To introduce a novel architecture for on-policy RL that constrains discrete policies to be unimodal.
• To leverage the continuity of underlying continuous action spaces through explicit unimodal probability distributions.
• To reduce variance in the policy gradient estimator and enhance learning stability.

Main Methods:

• Implementing a unimodal discrete policy architecture using Poisson probability distributions.
• Constraining the policy to be unimodal to better utilize the continuous nature of the action space.
• Conducting extensive experiments on challenging control tasks, including the Humanoid task.

Main Results:

• The unimodal discrete policy achieved significantly faster convergence compared to standard methods.
• Demonstrated higher performance in complex on-policy RL tasks, particularly in highly challenging scenarios like Humanoid.
• Theoretical analysis confirmed lower variance for the PG estimator with the proposed unimodal policy.

Conclusions:

• The proposed unimodal discrete policy architecture effectively addresses the variance issues associated with action space discretization in on-policy RL.
• This approach enhances learning stability and performance, especially in complex robotic control tasks.
• The explicit use of unimodal probability distributions better exploits the continuity of the action space for improved RL outcomes.