Fast Track Anything With Sparse Spatio-Temporal Propagation for Unified Video Segmentation

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Advanced "track-anything" models improve fine-grained video understanding.
• Existing models face challenges in robust and efficient temporal propagation for video segmentation.

Purpose of the Study:

• To propose a Sparse Spatio-Temporal Propagation (SSTP) method for robust and efficient unified video segmentation.
• To enhance temporal propagation by selectively leveraging key spatio-temporal features.

Main Methods:

• Designed a dynamic 3D spatio-temporal convolution for memory construction.
• Introduced a spatio-temporal aggregation reading strategy for efficient memory retrieval.
• Integrated SSTP with image segmentation foundation models like the segment anything model.

Main Results:

• Achieved state-of-the-art performance on five video segmentation tasks across eleven datasets.
• Outperformed both task-specific and unified methods in unified video segmentation.
• Demonstrated strong robustness in handling sparse, low-frame-rate videos.

Conclusions:

• SSTP offers a robust and efficient solution for unified video segmentation.
• The method effectively addresses data-scarce video segmentation tasks.
• SSTP is well-suited for real-world applications requiring reliable video tracking and segmentation.