Surgical Video Understanding with Alignment-Preserving Temporal Adaptation and Action Triplet Text Alignment

Area of Science:

• Medical image analysis
• Computer vision
• Artificial intelligence in surgery

Background:

• Surgical workflow understanding is crucial but hindered by the high cost and expertise required for dense video annotations.
• Existing methods struggle with annotation efficiency for long-horizon surgical videos.

Purpose of the Study:

• To develop a text-guided, annotation-efficient framework for surgical video understanding.
• To leverage a frozen surgical vision-language-pretrained (VLP) encoder with a lightweight temporal adapter.
• To improve surgical phase recognition and enable few-shot activity recognition.

Main Methods:

• Utilized a frozen SurgVLP image encoder for frame-level visual embeddings.
• Employed a lightweight temporal adapter to aggregate embeddings into clip-level representations.
• Evaluated using text-guided prototype matching for phase recognition and few-shot triplet recognition on the CholecT50 dataset.
• Implemented a Text Contrastive method with rich phase prompts.

Main Results:

• Temporal adaptation enhanced phase recognition while preserving the SurgVLP embedding space.
• The Text Contrastive method with rich phase prompts achieved superior phase recognition performance.
• The framework enabled classifier-free few-shot triplet recognition without a dedicated triplet classifier.

Conclusions:

• Effective surgical video understanding with limited annotations requires temporal adaptation and preserved alignment with the pretrained text space.
• Semantically informative text prompts are vital for improving performance.
• The proposed framework offers an annotation-efficient approach to surgical video analysis.