Self-Adaptive Vision-Language Tracking With Context Prompting.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Existing vision-language tracking methods struggle with modality gaps and the mismatch between static language and dynamic visual information.
• This performance limitation hinders the effective use of language semantics to improve tracking robustness.

Purpose of the Study:

• To propose a self-adaptive vision-language tracking framework that effectively bridges the modality gap.
• To enhance tracking robustness by enabling language features to dynamically evolve with visual context.

Main Methods:

• Leveraging the pre-trained multi-modal CLIP model for aligned visual-language representations.
• Introducing a context-aware prompting mechanism for dynamic adaptation of linguistic cues based on visual context.
• Employing a unified one-stream Transformer architecture for joint vision-only and vision-language tracking training.

Main Results:

• The proposed framework effectively bridges the modality gap and enhances tracking robustness.
• The large model achieved 55.0% AUC on LaSOT_EXT and 69.0% AUC on TNL2K.
• The language-only tracking model demonstrated performance comparable to state-of-the-art vision-only methods on TNL2K.

Conclusions:

• The self-adaptive framework successfully leverages language advantages to improve visual tracking.
• Dynamic adaptation of language embeddings to evolving visual context is key to enhanced robustness.
• The unified architecture supports versatile training scenarios, advancing vision-language tracking research.