TSMS-SAM2: Multi-scale Temporal Sampling Augmentation and Memory-Splitting Pruning for Promptable Video Object...

Area of Science:

• Computer Vision
• Medical Imaging
• Surgical Robotics

Background:

• Foundation models like Segment Anything Model 2 (SAM2) have advanced promptable video object segmentation and tracking (VOST).
• Applying VOST to surgical videos is difficult due to complex motion and memory redundancy in SAM2, hindering effective learning.
• Existing methods struggle with the dynamic and intricate nature of surgical environments.

Purpose of the Study:

• To enhance promptable VOST in surgical videos by developing a novel framework, TSMS-SAM2.
• To address challenges posed by rapid object motion and memory redundancy in SAM2 for surgical applications.
• To improve the robustness and efficiency of segmentation in complex surgical scenarios.

Main Methods:

• Proposed TSMS-SAM2 framework featuring multi-temporal-scale video sampling augmentation for motion variability.
• Implemented a memory splitting and pruning mechanism to organize and filter past frame features.
• Utilized EndoVis2017 and EndoVis2018 datasets for evaluation.

Main Results:

• TSMS-SAM2 achieved state-of-the-art performance with mean Dice scores of 95.24±0.96% on EndoVis2017 and 86.73±15.46% on EndoVis2018.
• Outperformed previous SAM-based and task-specific methods in surgical video segmentation.
• Ablation studies validated the effectiveness of multiscale temporal augmentation and memory splitting.

Conclusions:

• TSMS-SAM2 offers a robust and efficient solution for promptable VOST in complex surgical videos.
• The proposed framework significantly improves segmentation accuracy by tackling motion dynamics and memory redundancy.
• The findings highlight the potential of TSMS-SAM2 for advancing computer-assisted surgery and analysis.