Related Experiment Video
Updated: May 8, 2025
Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility
Published on: August 9, 2024
Automatic gesture recognition and evaluation in peg transfer tasks of laparoscopic surgery training
Shujun Ju1,2, Penglin Jiang2, Yutong Jin2
1West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China.
Background:
Laparoscopic surgery training is gaining increasing importance. To release doctors from the burden of manually annotating videos, we proposed an automatic surgical gesture recognition model based on the Fundamentals of Laparoscopic Surgery (FLS) and the Chinese Laparoscopic Skills Testing and Assessment (CLSTA) tools. Furthermore, statistical analysis was conducted based on a gesture vocabulary that had been designed to examine differences between groups at different levels.
Methods:
Based on the CLSTA, the training process of peg transfer can be represented by a standard sequence of seven surgical gestures defined in our gesture vocabulary. The dataset used for model training and testing included eighty videos recorded at 30 fps. All videos were rated by senior medical professionals from our medical training center. The dataset was processed using cross-validation to ensure robust model performance. The model applied is 3D ResNet-18, a convolutional neural network (CNN). An LSTM neural network was utilized to refine the output sequence.
Results:
The overall accuracy for the recognition model was 83.8% and the F1 score was 84%. The LSTM network improved model performance to 85.84% accuracy and an 85% F1 score. Every operative process starts with Gesture 1 (G1) and ends with G5, with wrong placement is labeled as G6. The average training time is 92 s (SD = 36). Variance was observed between groups for G1, G3, and G6, indicating that trainees may benefit from focusing their efforts on these relevant operations, while assisting doctors also in more effectively analyzing the training outcome.
Conclusion:
An automatic surgical gesture recognition model was developed for the peg transfer task. We also defined a gesture vocabulary along with the artificial intelligence model to sequentially describe the training operation. This provides an opportunity for artificial intelligence-enabled objective and automatic evaluation based on CLSTA in the clinic implementation.
