Surgical workflow recognition with temporal convolution and transformer for action segmentation.

Area of Science:

• Computer Vision
• Surgical Workflow Recognition
• Machine Learning in Healthcare

Background:

• Automatic surgical workflow recognition enhances surgeon training and enables context-aware systems for improved planning and decision-making.
• Utilizing temporal information is critical for context recognition in surgical procedures.
• Recurrent neural networks and transformers are current approaches for action recognition in surgical videos.

Purpose of the Study:

• To design and implement a novel two-stage method for surgical workflow recognition.
• To develop the Action Segmentation Temporal Convolutional Transformer (ASTCFormer) network for enhanced temporal awareness.
• To improve the accuracy and effectiveness of surgical workflow recognition systems.

Main Methods:

• A two-stage approach was employed, starting with R(2+1)D for video clip modeling.
• The Action Segmentation Temporal Convolutional Transformer (ASTCFormer) network was proposed for full video modeling in the second stage.
• ASTCFormer integrates action segmentation transformers (ASFormers) with temporal convolutional networks (TCNs) for temporally aware recognition.

Main Results:

• The ASTCFormer method demonstrated superior performance compared to recurrent neural networks, multi-stage TCN, and ASFormer.
• A relative improvement of [Formula: see text] in average segmental F1-score was achieved over the state-of-the-art ASFormer.
• The proposed method attained state-of-the-art results on the Cholec80 dataset, comprising 207 surgical videos.

Conclusions:

• Integrating temporal convolutional network (TCN) information into the ASFormer paradigm enhances the capture of temporal context.
• This integration leads to significant improvements in surgical workflow recognition accuracy.
• The ASTCFormer method represents aadvancement in automated surgical video analysis.