Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows.

Area of Science:

• Computer Science
• Medical Informatics
• Robotics

Background:

• Surgical workflow recognition is crucial for operating room efficiency and safety.
• Current methods often require extensive labeled data and focus on single tasks or modalities.
• Limitations include high annotation costs and lack of generalizability across diverse surgical procedures.

Purpose of the Study:

• To develop a novel semi-supervised learning approach for surgical workflow recognition.
• To leverage multimodal data (video and kinematics) and self-supervision for robust representation learning.
• To improve the efficiency of annotation while maintaining high performance in recognizing surgical gestures, phases, and steps.

Main Methods:

• A two-stage representation learning process was employed.
• Stage 1: Time contrastive learning for unsupervised spatiotemporal visual feature extraction from video.
• Stage 2: Multimodal Variational Autoencoder (VAE) fusion of visual and kinematic features, followed by recurrent neural networks for online recognition.

Main Results:

• The proposed method achieved performance comparable or superior to fully supervised models on JIGSAWS and MISAW datasets.
• Gesture recognition accuracy reached 83.3% on the JIGSAWS Suturing dataset.
• The model maintained high performance with significantly reduced annotation requirements (half the labels), demonstrating enhanced annotation efficiency.

Conclusions:

• The developed multimodal representation is versatile and effective across various surgical tasks.
• This approach significantly improves annotation efficiency for surgical workflow recognition models.
• The findings have substantial implications for advancing real-time decision-making systems in surgical environments.