Boosting Reinforcement Learning via Hierarchical Game Playing With State Relay

Area of Science:

• Robotics and Artificial Intelligence
• Machine Learning
• Motion Planning

Background:

• Deep Reinforcement Learning (DRL) is widely applied in motion planning.
• Current DRL methods reset agent states after task completion, leading to low sample utilization and limited environment exploration.
• Initial training stages in DRL exhibit weak learning abilities, impacting efficiency in complex tasks.

Purpose of the Study:

• To propose a novel Hierarchical Reinforcement Learning (HRL) framework called Hierarchical learning based on Game playing with State Relay (HGR).
• To enhance sample utilization, expand environment exploration, and improve training efficiency in complex motion planning tasks.

Main Methods:

• Introduced an auxiliary penalty to regulate task difficulty.
• Developed a state relay mechanism to utilize intermediate agent states and expand low-level policy exploration.
• Implemented the HGR framework for training agents in complex environments.

Main Results:

• The state relay mechanism effectively utilizes intermediate states, expanding environment exploration.
• The proposed HGR algorithm improves sample utilization rates and mitigates sparse reward issues.
• Simulation tests on MazeBase and MuJoCo platforms demonstrate significant performance enhancements.

Conclusions:

• HGR framework effectively addresses limitations in current DRL for motion planning.
• The state relay mechanism and auxiliary penalty contribute to improved sample efficiency and exploration.
• HGR offers a significant benefit to the broader reinforcement learning (RL) field.