Procedure-Aware Hierarchical Alignment for Open Surgery Video-Language Pretraining.

Area of Science:

• Artificial Intelligence in Surgery
• Computer Vision
• Surgical Data Science

Background:

• Surgical robotics and computer vision have advanced minimally invasive surgery but not open surgery.
• Open surgery, the predominant surgical approach, lacks large-scale datasets for AI development.
• Existing datasets are insufficient for training robust AI models for complex open surgical procedures.

Purpose of the Study:

• To introduce OpenSurgery, the largest video-text dataset for open surgery understanding.
• To develop and evaluate a novel framework, Hierarchical Surgical Knowledge Pretraining (HierSKP), for multimodal learning in open surgery.
• To establish a benchmark for evaluating AI performance in open surgery tasks.

Main Methods:

• Curated OpenSurgery dataset: 843 videos (102 hours) for pretraining and 400 videos (49 hours) for evaluation, annotated by expert surgeons.
• Proposed Hierarchical Surgical Knowledge Pretraining (HierSKP) framework utilizing granularity-aware contrastive learning.
• Incorporated hard negative sampling and Dynamic Time Warping (DTW)-based loss for enhanced temporal alignment and procedural understanding.

Main Results:

• OpenSurgery dataset is the largest publicly available resource for open surgery video-text pretraining and evaluation.
• HierSKP framework achieved state-of-the-art performance on the OpenSurgery-EVAL benchmark.
• Demonstrated superior performance in operation recognition, temporal action localization, and zero-shot cross-modal retrieval tasks.

Conclusions:

• The OpenSurgery dataset significantly advances the availability of resources for open surgery AI research.
• The HierSKP framework provides a robust method for multimodal representation learning in open surgery.
• This work paves the way for improved AI-driven assistance and understanding in open surgical procedures.