A new dataset and versatile multi-task surgical workflow analysis framework for thoracoscopic mitral valvuloplasty.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Medical Informatics

Background:

• Surgical Workflow Analysis (SWA) is crucial for AI-assisted surgery but under-researched for complex cardiac procedures.
• Existing SWA methods are mainly limited to laparoscopic surgeries, leaving thoracoscopic cardiac surgery largely unexplored.

Purpose of the Study:

• To introduce the first video dataset, TMVP-SurgVideo, for thoracoscopic cardiac mitral valvuloplasty (TMVP) workflow analysis.
• To propose SurgFormer, a novel AI framework for simultaneous recognition and anticipation of surgical phases and instruments in TMVP.
• To enable both offline and online inference for comprehensive SWA.

Main Methods:

• Developed TMVP-SurgVideo dataset with 57 videos and over 429K annotated frames.
• Proposed SurgFormer, a query-based Transformer framework with task-specific embeddings and an information interaction module.
• Incorporated intra-frame task-level and inter-frame temporal correlation learning within SurgFormer.
• Enabled dynamic memory bank for both offline and online inference without model modification.

Main Results:

• SurgFormer demonstrated effectiveness on the TMVP-SurgVideo dataset.
• Evaluated SurgFormer on the existing Cholec80 dataset, showing its versatility.
• The proposed framework successfully performs recognition and anticipation for surgical phases and instruments.

Conclusions:

• TMVP-SurgVideo is the first dataset enabling SWA research in thoracoscopic cardiac mitral valvuloplasty.
• SurgFormer provides a comprehensive and effective AI solution for SWA in TMVP, outperforming limitations of existing methods.
• The developed dataset and framework advance AI-assisted intelligent surgery for complex cardiac procedures.