An online trajectory guidance framework via imitation learning and interactive feedback in robot-assisted surgery

Improving the manipulation performance of surgical instruments is important for novice surgeons, as it directly affects the safety and outcome of robot-assisted surgery. To reduce the difference between expert and novice surgeons, learning the instrument movement trajectories generated by experts is an effective approach for novices to foster their muscle memory and improve manipulation skills. In this work, we propose an online trajectory guidance framework to generate expert-like movement trajectories so that novice surgeons can receive intra-operative trajectory guidance to achieve a similar manipulation performance as experts. First, Dynamic Movement Primitives (DMP) based Imitation Learning (IL) is implemented to model the 3D trajectories demonstrated by experts for adaptive trajectory generation at different start and end points. To introduce the obstacle avoidance capability into IL, we propose a vision-based strategy involving stereo reconstruction, object detection and segmentation to recover the 3D information of obstacles so that they can be coupled into DMP as an obstacle avoidance term. Furthermore, we introduce Augmented Reality (AR) and Interactive Feedback (IF) including visual and force feedback to enhance the trajectory reproduction accuracy of novice surgeons during operation. The experiment was conducted based on a 3D peg-transfer task in two different scenes (with changed start and end points, and with the obstacle present) using a standard da Vinci Research Kit robot. Ten non-expert human subjects were invited to evaluate the online trajectory guidance framework by reproducing the expert-like manipulation trajectories, and the experimental results showed that the novices with the assistance of AR and IF achieved promising trajectory reproduction performance (the mean distance error E_mean was reduced by 76.47% and 65.15% in two different intra-operative scenes, respectively), narrowing the manipulation gap with experts.