Parameter-exploring policy gradients.

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Partially observable Markov decision problems (POMDPs) present significant challenges for reinforcement learning.
• Traditional policy gradient methods often suffer from high variance gradient estimates.
• Model-free approaches are desirable for real-world applications where system dynamics are unknown.

Purpose of the Study:

• To introduce a novel model-free reinforcement learning method for POMDPs.
• To reduce gradient variance compared to standard policy gradient techniques.
• To demonstrate superior performance on complex control tasks.

Main Methods:

• Parameter space sampling for likelihood gradient estimation.
• A novel model-free reinforcement learning algorithm designed for POMDPs.
• Comparative analysis against standard policy gradients, finite difference, and population-based methods.

Main Results:

• The proposed method yields lower variance gradient estimates.
• Outperformance observed in complex control tasks, including humanoid robot locomotion.
• Performance gains are maximized with symmetric parameter sampling.

Conclusions:

• The developed method offers a significant improvement for model-free reinforcement learning in POMDPs.
• Symmetric sampling is a key factor in enhancing performance.
• Component analysis validates the effectiveness of individual method elements.