Single-Frame Supervision for Spatio-Temporal Video Grounding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Spatio-Temporal Video Grounding (STVG) localizes objects in videos using natural language queries.
• Current weakly supervised methods for STVG suffer from performance degradation due to labor-intensive tube annotations.

Purpose of the Study:

• To develop a less expensive STVG method with acceptable accuracy using single-frame supervision.
• To address the performance gap in weakly supervised STVG.

Main Methods:

• Proposed Two-Stage Multiple Instance Learning (T-SMILE) method.
• Utilizes single-frame bounding box annotations to create pseudo-labels for training.
• Incorporates multiple instance learning for timestamp recognition, spatial prior constraints for feature learning, and curriculum learning for branch adaptation.

Main Results:

• T-SMILE significantly outperforms existing weakly supervised STVG methods.
• Achieved performance comparable to or better than some fully supervised methods despite lower annotation costs.
• Introduced a large-scale benchmark for single-frame annotated STVG.

Conclusions:

• Single-frame supervision is a viable paradigm for efficient STVG.
• T-SMILE offers a promising solution for accurate and cost-effective STVG.
• The new benchmark will facilitate future research in weakly supervised STVG.