Accelerating SAM2 with Efficient Memory Attention Module via Spatiotemporal Token Pruning

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Segment Anything Model (SAM) is a foundation model for image segmentation.
• SAM2 extends SAM for video processing using a memory attention module.
• Transformer-based modules propagate temporal information via cross-attention.

Purpose of the Study:

• To accelerate the inference time of SAM2 for video segmentation.
• To address the computational complexity of the attention mechanism in SAM2.
• To reduce the sequence length of memory tokens without significant performance degradation.

Main Methods:

• Proposed a spatiotemporal token pruning method to reduce memory token sequence length.
• Leveraged findings of high attention sparsity and accumulation in memory tokens.
• Developed a pruning strategy based on boundary delineation and temporal trackability for video dense-prediction.
• Utilized three scoring functions to create a reference matrix evaluating memory token importance.
• Applied scoring functions layer-wise to reflect spatial or temporal information propagation.

Main Results:

• Achieved effective compression of up to 87% of memory tokens.
• Accelerated per-frame inference time by 1.7×.
• Caused negligible disturbance to the original model performance across benchmark datasets.
• Demonstrated a non-retraining approach for efficient video segmentation.

Conclusions:

• The proposed spatiotemporal token pruning method significantly enhances SAM2's efficiency.
• This approach provides a viable solution for real-time video segmentation tasks.
• Efficient video segmentation is crucial for various downstream applications.