Segment Anything for Video: A Comprehensive Review of Video Object Segmentation and Tracking from Past to Future

Video Object Segmentation and Tracking (VOST) presents a complex yet critical challenge in computer vision, requiring robust integration of segmentation and tracking across temporally dynamic frames. Traditional methods have struggled with domain generalization, temporal consistency, and computational efficiency. The emergence of foundation models like the Segment Anything Model (SAM) and its successor, SAM2, has introduced a paradigm shift, enabling prompt-driven segmentation with strong generalization capabilities. Building upon these advances, this survey offers a comprehensive review of SAM-based methods for VOST, structured along three temporal dimensions: past, present, and future. We examined strategies for retaining and updating historical information (past), approaches for extracting and optimizing discriminative features from the current frame (present), and motion prediction and trajectory estimation mechanisms for anticipating object dynamics in subsequent frames (future). In doing so, we highlighted the evolution from early memory-based architectures to the streaming memory and real-time segmentation capabilities of SAM-based approaches. We also discussed recent innovations, such as motion-aware memory selection and trajectory-guided prompting, that aim to enhance both accuracy and efficiency. Finally, we identified several open challenges, including efficient memory feature updating, support for open-vocabulary and multi-modal settings, and prior-guided motion estimation for real-time inference, aiming for robust handling of fast motion, motion blur, occlusions, visually similar objects, and cross- or multi-modal VOST scenarios. We also conclude by outlining promising directions for future research to advance long-term and real-time video object segmentation. This survey provides a timely and structured overview of the field, aiming to guide researchers and practitioners in advancing the state of VOST through the lens of foundation models. We provide a public website compiling representative works, datasets, and download links: https://github.com/apple1986/sam_vost.