Learning Modality Complementary Features with Mixed Attention Mechanism for RGB-T Tracking.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Sensor Fusion

Background:

• Single-modality tracking struggles with varying environmental conditions.
• Integrating visible (RGB) and thermal (T) data offers potential for enhanced robustness.
• Existing fusion methods may not optimally adapt to modality dominance shifts.

Purpose of the Study:

• To develop an adaptive RGB-T tracking algorithm that synergistically fuses visible and thermal data.
• To improve tracking accuracy and robustness by leveraging the dominant modality.
• To introduce a mixed-attention mechanism for effective feature extraction and fusion.

Main Methods:

• Utilized separate transformer backbone branches for modality-specific and shared feature extraction.
• Implemented mixed-attention operations within the backbone for template-search image interaction.
• Designed a modality shared-specific feature interaction structure with mixed attention for fusion.

Main Results:

• The proposed MACFT tracker demonstrated superior performance over existing RGB-T trackers on public datasets.
• The mixed-attention mechanism effectively suppressed noise from low-quality modalities.
• The tracker successfully enhanced information from the dominant modality.
• Achieved robust performance in general and long-term tracking scenarios.

Conclusions:

• The MACFT tracker effectively achieves complementary fusion of RGB and thermal data.
• Adaptive modality leveraging via mixed attention leads to enhanced tracking robustness.
• The proposed method offers a significant advancement in RGB-T visual tracking.