Kernel dynamic policy programming: Applicable reinforcement learning to robot systems with high dimensional states.

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Model-free reinforcement learning in high-dimensional Markov decision processes faces challenges with brittleness and computational complexity.
• Existing value function approaches struggle to scale effectively to systems with numerous state dimensions.

Purpose of the Study:

• To introduce a new value function approach for reinforcement learning applicable to high-dimensional state spaces.
• To enhance the applicability of value function methods in complex robotic systems.

Main Methods:

• Proposed Kernel Dynamic Policy Programming (KDPP), a novel algorithm for model-free reinforcement learning.
• KDPP utilizes Kullback-Leibler divergence for smooth value function updates, stabilizing learning.
• Employs the kernel trick for efficient value function approximation in high-dimensional state spaces.

Main Results:

• KDPP demonstrated superior performance in a simulated n-DOF manipulator reaching task, learning a viable policy at n=40.
• Successfully applied KDPP to a real-world robotic hand (32-dimensional state space) for unscrewing a bottle cap.
• Achieved efficient learning with limited samples and ordinary computing resources on the robotic system.

Conclusions:

• KDPP effectively addresses computational complexity and brittleness in high-dimensional reinforcement learning.
• The algorithm enables practical application of value function-based reinforcement learning to complex robotic systems.
• KDPP represents a significant advancement for reinforcement learning in real-world, high-dimensional applications.