Towards Universal Modal Tracking With Online Dense Temporal Token Learning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Multi-modal tracking enhances robustness by integrating diverse sensor data (e.g., RGB, thermal, depth, event).
• Existing multi-modal trackers often require task-specific architectures and extensive, independent training.
• A unified approach for various tracking modalities remains a significant challenge.

Purpose of the Study:

• To develop a universal video-level modality-aware tracking model (UM-ODTrack) adaptable to multiple tracking tasks.
• To enable a single model architecture and parameter set to handle RGB, RGB+Thermal, RGB+Depth, and RGB+Event tracking.
• To improve tracking performance and reduce training complexity through novel temporal token learning and cross-modal fusion.

Main Methods:

• Video-level sampling to capture broader temporal context.
• Online dense temporal token association for appearance and motion propagation.
• Gated perceivers with attention mechanisms for adaptive cross-modal representation learning.
• One-shot training for modality-scalable multi-task inference.

Main Results:

• UM-ODTrack achieves state-of-the-art (SOTA) performance on visible and multi-modal benchmarks.
• The model effectively leverages previous frame information as temporal prompts for future inference.
• The one-shot training scheme significantly reduces training burden while enhancing model representation.

Conclusions:

• UM-ODTrack offers a unified and efficient solution for diverse video tracking tasks.
• The proposed method demonstrates superior performance and generalization capabilities across different modalities.
• This work advances the field of multi-modal visual tracking with a scalable and effective approach.