Related Experiment Video
Updated: Aug 30, 2026
Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality
Published on: August 9, 2024
TBDM: Temporal boundary distillation module for surgical gesture segmentation
Ezem Sura Ekmekci1, Sebastien Frey2,3, Snehashis Majhi4
1Université Côte d'Azur, Inria, Sophia-Antipolis, France. ezem-sura.ekmekci@inria.fr.
Purpose:
Achieving fine-grained understanding of surgical gestures remains a fundamental challenge in computer vision, due to the subtle and temporally overlapping nature of surgical motions. Gesture boundaries, where transitions between surgical actions occur, present challenges for precise temporal localization. We propose a temporal boundary analysis framework that improves overall surgical gesture segmentation by explicitly modeling transitions between actions. While most existing methods rely on both RGB and kinematic data, our approach operates on RGB-only video, without requiring additional annotations or computational overhead at inference.
Methods:
We introduce a temporal boundary distillation module (TBDM) that leverages privileged information during training to learn boundary-aware features. TBDM employs cross-attention between class-present and class-absent temporal regions derived from ground-truth annotations, explicitly encoding transition information. A lightweight projection layer learns boundary-aware features through knowledge distillation from TBDM, supervised by classification and distillation loss (MSE). At inference, only the trained projection layer is required, resulting in no additional computational cost.
Results:
We evaluated TBDM on CholecT50 and RARP-45 surgical datasets. TBDM consistently improved baseline models across all metrics, achieving up to +8.5 edit score improvement on CholecT50. On RARP-45, our approach achieved state-of-the-art edit score (81.4) and F1@50 (77.9), demonstrating effectiveness across different architectures and datasets.
Conclusion:
TBDM provides a generalized, plug-and-play framework for fine-grained surgical gesture segmentation, using RGB-only data. By explicitly modeling temporal boundaries, it achieves consistent improvement across multiple architectures and surgical datasets without increasing inference complexity. Code is publicly available at https://github.com/ezemsuraekmekci/TBDM-Surgical-Gesture-Segmentation .
