Deep reinforcement learning coupled with musculoskeletal modelling for a better understanding of elderly falls

Reinforcement learning (RL) has been used to study human locomotion learning. One of the current challenges in healthcare is our understanding of and ability to slow the decline due to muscle ageing and its effect on human falls. The purpose of this study was to investigate reinforcement learning for human movement strategies when modifying muscle parameters to account for age-related changes. In particular, human falls with modified physiological factors were modelled and simulated to determine the effect of muscle descriptors for ageing on kinematic behaviour and muscle force control. A 3D musculoskeletal model (8 DoF and 22 muscles) of the human body was used. The deep deterministic policy gradient (DDPG) method was implemented. Different muscle descriptors for ageing were integrated, including changes in maximum isometric force, contraction velocity, the deactivation time constant and passive muscle strain. Additionally, the effects of isometric force reductions of 10, 20 and 30% were also considered independently. An environment for the simulation was developed using the opensim-rl package for Python with the training process completed on Google Compute Engine. The simulation outcomes for healthy young adult and elderly falls under modified muscle behaviours were compared to experimental observations for validation. The result of our elderly simulation for multiple ageing-related factors (M_all) produced a walking speed of 0.26 m/s for the two steps taken prior to the fall. The over activation of the hip extensors and inactivation of knee extensors led to a backward fall for this elderly simulation. The inactivated rectus femoris and right tibialis are main actors of the forward fall. Our simulation outcomes are consistent with experimental observations through the comparison of kinematic features and motion history evolution. We showed in the present study, for the first time, that RL can be used as a strategy to explore the effect of ageing muscle physiological factors on kinematics and muscle control during falls. Our findings show that the elderly fall model for the M_all condition more closely resembles experimental elderly fall data than our simulations which considered age-related reductions of force alone. As future perspectives, the behaviour preceding a fall will be studied to establish the strategies used to avoid falls or fall with minimal consequence, leading to the identification of patient-specific rehabilitation programmes for elderly people.