Human locomotion with reinforcement learning using bioinspired reward reshaping strategies

Area of Science:

• Robotics and Artificial Intelligence
• Biomechanics and Motor Control
• Computational Neuroscience

Background:

• Reinforcement learning (RL) shows promise for advancing artificial intelligence, but challenges remain in applying it to complex systems like human musculoskeletal models for dynamic movement.
• Existing research often lacks biomechanical perspectives and the integration of biological knowledge for developing motor control models, particularly for pathological conditions, and rarely employs reward reshaping.

Purpose of the Study:

• To design and evaluate novel bioinspired reward function strategies for human locomotion learning within a reinforcement learning (RL) framework.
• To investigate the application of the Deep Deterministic Policy Gradient (DDPG) method for controlling a 3D musculoskeletal model in simulated walking and falling scenarios.

Main Methods:

• Utilized a 3D musculoskeletal model (8 Degrees of Freedom, 22 muscles) of a healthy adult.
• Developed a virtual interactive environment using the opensim-rl library for simulation.
• Defined three distinct reward functions for walking, forward falls, and side falls, training the model using Google Cloud Compute Engine.

Main Results:

• Simulated models achieved walking distances of 18 to 20.5 meters, revealing compensatory muscle activation strategies.
• Identified key muscles (Soleus, Tibia Anterior, Vastii) involved in forward falls, with increased activation post-fall.
• Observed intensive muscle activation on the expected fall side during side falls to induce unbalancing, with all kinematic and muscle patterns consistent with experimental data.

Conclusions:

• The integration of computational resources, biomechanical knowledge, and human expertise is crucial for developing robust RL solutions for human locomotion.
• The study successfully demonstrated the efficacy of bioinspired reward functions in enabling a musculoskeletal model to learn complex locomotion and fall behaviors.
• Future work will extend to larger parameter spaces and explore stochastic RL models to address uncertainties in musculoskeletal dynamics for general artificial intelligence in locomotion.