SF-TMN: SlowFast temporal modeling network for surgical phase recognition

Area of Science:

• Computer Vision
• Medical Image Analysis
• Surgical Education Technology

Background:

• Automatic surgical phase recognition is vital for objective assessment in surgical training.
• Current methods often rely on frame-level features for temporal modeling, potentially missing broader temporal dynamics.
• Effective temporal modeling is key to accurately recognizing surgical phases from video data.

Purpose of the Study:

• To develop a novel network, the SlowFast temporal modeling network (SF-TMN), for enhanced offline surgical phase recognition.
• To enable both frame-level and segment-level temporal modeling within a unified framework.
• To improve the accuracy and robustness of surgical phase recognition systems.

Main Methods:

• Proposed the SlowFast temporal modeling network (SF-TMN) integrating frame and segment-level temporal modeling.
• Utilized a pretrained feature extraction network for generating frame features.
• Employed a Slow Path for frame-level temporal modeling and a Fast Path for segment-level temporal modeling.
• Explored MS-TCN and ASFormer as backbone temporal modeling networks.

Main Results:

• SF-TMN achieved state-of-the-art performance on Cholec80 and Cataract-101 surgical phase recognition datasets.
• The SF-TMN with ASFormer backbone surpassed the previous state-of-the-art by ~1% in accuracy and ~1.5% in recall on Cholec80.
• Achieved state-of-the-art results on action segmentation benchmarks: 50salads, GTEA, and Breakfast.

Conclusions:

• Combining frame-level and segment-level temporal information significantly benefits surgical phase recognition.
• Temporal refinement stages enhance the modeling of surgical phases.
• The SF-TMN framework offers a flexible and effective approach for video-based surgical analysis.