Deep Reinforcement Learning for Real-World Humanoid Robot Locomotion Control with Automatic Reward Learning

Area of Science:

• Robotics
• Artificial Intelligence
• Control Systems

Background:

• Humanoid robots are complex systems requiring advanced motion control for diverse applications.
• Deep Reinforcement Learning (DRL) offers adaptive control but faces challenges in reward function design for real-world tasks.
• Manual reward engineering for DRL is time-consuming, requires expertise, and can lead to suboptimal performance or mission failure.

Purpose of the Study:

• To develop an automatic reward learning method for DRL-based humanoid robot locomotion control.
• To address the limitations of manual reward function design in DRL.
• To improve the efficiency and robustness of humanoid robot motion control.

Main Methods:

• A bilevel optimization framework was developed for automatic reward learning.
• The upper level adaptively constructs and optimizes the reward function.
• The lower level employs DRL to learn the locomotion control policy using the learned reward function.

Main Results:

• The proposed method significantly improved learning efficiency and performance compared to manual reward functions.
• Experiments demonstrated effectiveness in simulation (MuJoCo, Isaac Lab) and real-world deployment (Unitree G1 robot).
• Successful sim-to-real transfer of control policies was achieved, enhancing deployment capabilities.

Conclusions:

• Automatic reward learning offers a more efficient and effective approach for DRL in humanoid robot locomotion.
• The method accelerates the development and deployment of stable, adaptive humanoid robots.
• This work provides a promising pathway for advancing practical humanoid robot applications.