Augmenting efficient real-time surgical instrument segmentation in video with point tracking and Segment Anything

Area of Science:

• Computer Vision
• Medical Robotics
• Artificial Intelligence

Background:

• The Segment Anything model (SAM) offers powerful segmentation but faces limitations in robotic surgery due to high computational costs and per-frame prompting.
• Clinical applications like augmented reality guidance demand minimal user intervention and efficient inference.

Purpose of the Study:

• To develop an efficient and generalizable surgical instrument segmentation framework for robotically assisted surgery.
• To address the computational and prompting limitations of SAM in surgical contexts.

Main Methods:

• Adopted lightweight SAM variants and fine-tuning for surgical scenes.
• Integrated an online point tracker with the fine-tuned SAM to prompt segmentation using sparse points.
• Ensured temporal consistency by tracking points across video frames, even with occlusions or departures from the field of view.

Main Results:

• Achieved superior performance compared to the state-of-the-art semi-supervised method XMem on the EndoVis 2015 dataset (84.8 IoU, 91.0 Dice).
• Demonstrated comparable performance to fully supervised methods on ex vivo and in vivo surgical datasets.
• Showcased strong zero-shot generalization on the label-free STIR dataset.
• Attained inference speeds exceeding 25 FPS on a GeForce RTX 4060 and 90 FPS on a 4090 GPU.

Conclusions:

• The proposed framework effectively enhances SAM's efficiency and generalization for surgical instrument segmentation.
• The method shows significant potential for real-time applications in robotically assisted surgery, including augmented reality guidance.
• The combination of lightweight SAM and point tracking offers a promising direction for advancing surgical computer vision.