Reinforcement Q-Learning Control With Reward Shaping Function for Swing Phase Control in a Semi-active Prosthetic...

Area of Science:

• Biomedical Engineering
• Robotics
• Artificial Intelligence

Background:

• Semi-active prosthetic knees require advanced control systems for natural gait.
• Magnetorheological dampers offer tunable resistance for prosthetic knee control.
• Reinforcement learning (RL) presents a promising approach for adaptive control.

Purpose of the Study:

• To develop and evaluate a novel reinforcement learning (RL) control algorithm for a semi-active prosthetic knee.
• To design a reward shaping function for optimizing prosthetic knee control based on subject-specific gait data.
• To compare the proposed RL control strategy against conventional methods and existing advanced algorithms.

Main Methods:

• Model-free reinforcement Q-learning was employed for voltage control of a magnetorheological damper.
• A reward function was engineered using a performance index based on subject-specific knee angle trajectories.
• The control algorithm was trained and validated on diverse walking speed datasets.

Main Results:

• The proposed reward shaping function outperformed a conventional single reward function in terms of normalized root mean squared error.
• Faster convergence trends were observed with the novel reward function.
• The RL control strategy demonstrated adaptability to multiple walking speeds and achieved desired performance indices.
• Overall performance surpassed user-adaptive control and showed competitive results against neural network predictive control.

Conclusions:

• The developed reinforcement learning control algorithm with a novel reward shaping function offers superior performance for semi-active prosthetic knees.
• The strategy effectively adapts to varying walking speeds, enhancing gait naturalness and user experience.
• This approach represents a significant advancement in prosthetic knee control technology.