Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Medical Informatics

Background:

• Automatic surgical phase recognition is crucial for operating room efficiency and surgical video analysis.
• Current methods leverage spatial and temporal data, but advancements are needed for enhanced accuracy.

Purpose of the Study:

• To develop novel Transformer-based models for accurate surgical phase recognition.
• To improve the modeling of temporal information at multiple scales for better video analysis.

Main Methods:

• Proposed Multi-Scale Action Segmentation Transformer (MS-AST) for offline and MS-ASCT for online recognition.
• Utilized ResNet50 or EfficientNetV2-M for spatial feature extraction.
• Implemented multi-scale temporal self-attention and cross-attention mechanisms.

Main Results:

• Achieved 95.26% (online) and 96.15% (offline) accuracy on the Cholec80 dataset.
• Established new state-of-the-art results for surgical phase recognition.
• Demonstrated superior performance on non-medical video action segmentation datasets.

Conclusions:

• The proposed MS-AST and MS-ASCT models significantly advance surgical phase recognition.
• These models offer enhanced capture of temporal relationships, leading to state-of-the-art performance.
• The approach is effective for both medical and general video action segmentation tasks.