Spatial-Temporal Scene Graph Generation for Open-Vocabulary Multiple Object Tracking.

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Open-vocabulary multiple object tracking (MOT) faces challenges in data association, especially with occlusions and ambiguous appearances.
• Existing MOT methods often rely on appearance cues, which are unreliable in complex scenarios.
• Advances in vision-language models aid object classification but not data association in open-vocabulary MOT.

Purpose of the Study:

• To propose a novel open-vocabulary MOT method, SSGTrack, that enhances data association.
• To introduce a Spatial-temporal Scene Graph (SSG) for capturing object relationships across frames.
• To develop a Context-aware Contrastive Learning (CCL) strategy for improved discriminative representation learning.

Main Methods:

• Constructing a Spatial-temporal Scene Graph (SSG) by extracting contextual information from Transformer decoder's self-attention layers.
• Representing detected objects as nodes and frame-level connectivity as edge weights in the SSG.
• Implementing Context-aware Contrastive Learning (CCL) by using differing background features as negative samples to enhance representation learning.

Main Results:

• SSGTrack demonstrates superior tracking performance on challenging MOT benchmarks.
• The proposed SSG effectively captures semantic and spatial relationships between objects.
• CCL strategy enhances the model's ability to differentiate visually similar objects and background distractors, improving robustness.

Conclusions:

• SSGTrack offers a novel and effective approach to data association in open-vocabulary MOT.
• The integration of SSG and CCL significantly improves tracking accuracy and robustness in complex scenarios.
• This method advances the state-of-the-art in open-vocabulary multiple object tracking.