Learning agile soccer skills for a bipedal robot with deep reinforcement learning

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Developing sophisticated movement skills for robots is challenging.
• Manual design of complex behaviors is often impractical.
• Deep reinforcement learning (deep RL) offers a potential solution.

Purpose of the Study:

• To investigate deep RL for synthesizing complex and safe movement skills in miniature humanoid robots.
• To train a robot for a one-versus-one soccer game using deep RL.
• To assess the transferability of learned skills from simulation to real robots.

Main Methods:

• Trained a deep RL agent in simulation for a simplified soccer game.
• Utilized high-frequency control, dynamics randomization, and perturbations for robust training.
• Transferred learned policies to a real miniature humanoid robot with zero-shot learning.

Main Results:

• The agent learned dynamic skills: fall recovery, walking, turning, and kicking.
• Achieved significantly improved performance: 181% faster walking, 302% faster turning, 63% less recovery time, 34% faster kicking.
• Demonstrated adaptive tactical behavior and anticipation of game events.
• Successful zero-shot transfer from simulation to the real robot.

Conclusions:

• Deep reinforcement learning can effectively synthesize complex and safe movement skills for humanoid robots.
• The trained agent exhibited superior performance and adaptability compared to scripted baselines.
• The sim-to-real transfer was successful due to specific training techniques, enabling practical applications.