Cross-modal self-supervised representation learning for gesture and skill recognition in robotic surgery.

Area of Science:

• Robotics
• Machine Learning
• Computer Vision

Background:

• Multi- and cross-modal learning integrate diverse data for comprehensive understanding.
• Synchronized data streams in cross-modal learning serve as effective self-supervisory signals.
• Self-supervised continual learning in surgical robotics promises adaptive, lifelong learning capabilities.

Purpose of the Study:

• To develop a self-supervised learning paradigm for surgical robotics using synchronous video and kinematics.
• To enable lifelong learning systems that adapt to different surgeons and surgical cases.
• To achieve a more generalized machine understanding of surgical processes.

Main Methods:

• Utilized an encoder-decoder network to map optical flow from video to kinematics sequences.
• Employed clustering on latent representations to identify surgeon gestures and skill levels.
• Demonstrated generalizability by classifying skills and gestures on unseen tasks within the JIGSAWS dataset.

Main Results:

• Achieved 59-70% accuracy in classifying surgical gestures for trained tasks.
• Reported 45-65% accuracy for classifying gestures on tasks not included in training.
• Observed that unseen gestures formed distinct clusters in the latent space, suggesting potential for novel interaction identification.

Conclusions:

• Optical flow representations derived from predicting kinematics are effective even for novel surgical tasks.
• The self-supervised learning paradigm shows immediate utility across various tasks.
• This approach facilitates future research in lifelong and user-specific surgical robotics learning.