Cross-Modal Object Tracking via Modality-Aware Fusion Network and a Large-Scale Dataset

Area of Science:

• Computer Vision
• Machine Learning
• Image Processing

Background:

• Visual object tracking using only RGB data struggles with low-light conditions and invalid targets.
• Existing multimodal tracking solutions are often complex and lack practical applicability.
• Near-infrared (NIR) imaging offers an alternative but presents challenges in tracking across heterogeneous RGB and NIR modalities.

Purpose of the Study:

• To develop an adaptive cross-modal object tracking algorithm to address challenges in tracking across RGB and NIR modalities.
• To propose a flexible and efficient network that integrates information from both RGB and NIR data.
• To introduce a comprehensive benchmark dataset for advancing cross-modal object tracking research.

Main Methods:

• Proposed the modality-aware fusion network (MAFNet) featuring an adaptive weighting module and a modality-specific representation module.
• MAFNet dynamically adjusts the contribution of RGB and NIR modalities using predicted fusion weights.
• Developed the CMOTB benchmark dataset with 1000 video sequences (over 799K frames) across 61 categories.

Main Results:

• MAFNet effectively bridges the appearance gap between RGB and NIR modalities, enabling modality-aware target representation.
• The proposed algorithm demonstrates superior performance compared to state-of-the-art methods in cross-modal object tracking.
• The CMOTB dataset provides extensive data to validate the effectiveness of cross-modal tracking algorithms.

Conclusions:

• MAFNet offers a simple, effective, and efficient solution for cross-modal object tracking, outperforming existing methods.
• The CMOTB dataset and MAFNet provide a strong foundation for future research in cross-modal object tracking.
• Publicly available dataset, toolkit, and code will facilitate further advancements in the field.