Humanoids Learning to Walk: A Natural CPG-Actor-Critic Architecture.

Area of Science:

• Robotics
• Computational Neuroscience
• Artificial Intelligence

Background:

• Humanoid locomotion learning is challenging.
• Dynamic Systems Theory (DST) offers a novel approach via environmental interaction.
• Reinforcement Learning (RL) provides adaptive linking through reward systems.

Purpose of the Study:

• Propose an integrated model combining DST and RL for humanoid robot walking.
• Apply the model to a NAO robot learning to walk.
• Demonstrate emergent walking ability from value-based environmental interaction.

Main Methods:

• Integrated a simplified central pattern generator (CPG) architecture with an actor-critic RL approach (cpg-actor-critic).
• Employed least-square-temporal-difference learning with natural gradient for fast convergence and balanced exploration/exploitation.
• Utilized a dynamic value function as an adaptive stability indicator instead of traditional rewards.

Main Results:

• The cpg-actor-critic model successfully enabled a NAO robot to learn walking.
• The emergent walking ability stemmed from value-based interaction with the environment.
• Analysis using a DST-based embodied cognition approach provided insights into the learning process.

Conclusions:

• Robot walking emerges from integrating sensorimotor activity and value.
• The proposed model effectively combines DST, RL, and neuroscientific principles.
• This approach offers a promising direction for humanoid robot learning and embodied cognition research.