Unified Multi-Modal Object Tracking Through Spatial-Temporal Propagation and Modality Synergy

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Multi-modal object tracking (MMOT) leverages multiple sensors to overcome limitations of single-sensor systems.
• Existing MMOT methods struggle with data heterogeneity, inter-modal imbalance, and degraded reliability in complex, dynamic environments.

Purpose of the Study:

• To propose SMUTrack, a unified framework for multi-modal object tracking that addresses challenges in representation learning and generalization.
• To enhance multi-modal consistent perception and data aggregation in dynamic scenarios.

Main Methods:

• Developed SMUTrack, a unified framework with global shared parameters integrating three downstream MMOT tasks.
• Implemented a batch merging-and-splitting strategy with multi-task joint training to establish cross-modal correlations.
• Introduced a hierarchical modality synergy and reinforcement (HMSR) module and a gated fusion and context awareness (GFCA) module for progressive information exchange.
• Incorporated a spatial-temporal information propagation (SIP) mechanism to learn trajectory and appearance cues for contextual relationships.

Main Results:

• SMUTrack demonstrates outstanding performance on mainstream MMOT datasets.
• The framework exhibits powerful adaptability across various MMOT tasks.
• The proposed modules effectively improve multi-modal representation and tracking robustness.

Conclusions:

• SMUTrack provides a unified and effective solution for multi-modal object tracking.
• The framework successfully addresses data heterogeneity and dynamic environment challenges.
• SMUTrack offers a robust and adaptable approach for advanced MMOT applications.