Efficient Reinforcement Learning via Probabilistic Trajectory Optimization.

Area of Science:

• Robotics
• Machine Learning
• Control Theory

Background:

• Reinforcement learning (RL) in continuous spaces presents challenges for sample efficiency and policy optimization.
• Model-based RL methods often rely on policy parameterization, limiting their flexibility.
• Existing approaches may struggle with complex dynamics and require extensive data.

Purpose of the Study:

• To develop a trajectory optimization approach for reinforcement learning in continuous state and action spaces.
• To create a method that learns time-varying control policies without explicit policy parameterization.
• To enhance learning speed and data efficiency in complex control tasks.

Main Methods:

• Probabilistic Differential Dynamic Programming (PDDP) represents system dynamics using Gaussian processes (GPs).
• Iterative local dynamic programming is performed in Gaussian belief spaces around a nominal trajectory.
• The method learns policies via successive forward-backward sweeps, incorporating prior knowledge and enabling risk-sensitive learning.

Main Results:

• PDDP converges globally to a stationary point under specific conditions, as shown by convergence analysis.
• The framework effectively incorporates prior model knowledge to accelerate learning.
• Demonstrated effectiveness and efficiency on nontrivial tasks, outperforming a state-of-the-art GP-based policy search method.

Conclusions:

• PDDP offers a superior combination of learning speed, data efficiency, and applicability compared to existing methods.
• The approach provides a robust framework for reinforcement learning in continuous control problems.
• PDDP advances the state-of-the-art in model-based reinforcement learning for complex systems.