Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Multistep robotic manipulation tasks, like block stacking, are challenging due to the need for hierarchical planning and control.
• Current reinforcement learning (RL) methods struggle with long-horizon tasks, varied outcomes, and efficient exploration, especially with sparse rewards.

Purpose of the Study:

• To develop a unified hierarchical reinforcement learning framework to address limitations in current RL for complex robotic manipulation.
• To enable agents to learn varied outcomes in multistep tasks and improve learning efficiency.

Main Methods:

• Introduced the Universal Option Framework (UOF), a hierarchical reinforcement learning approach.
• Developed parallel training for symbolic planning and kinematic control policies.
• Implemented an auto-adjusting exploration strategy (AAES) for low-level control stabilization.
• Utilized abstract demonstrations for high-level policy acceleration.

Main Results:

• The UOF framework successfully accomplished multistep block-stacking tasks with diverse block configurations and robot degrees of freedom.
• The proposed method demonstrated increased efficiency and stability in learning complex manipulation tasks.
• Significantly reduced memory consumption compared to existing approaches.

Conclusions:

• The Universal Option Framework provides an effective solution for complex, multistep robotic manipulation tasks.
• UOF enhances learning efficiency and enables robots to achieve varied outcomes through hierarchical planning and control.
• The framework shows promise for advancing autonomous robotic capabilities in intricate manipulation scenarios.