Revisiting RGBT Tracking Benchmarks From the Perspective of Modality Validity: A New Benchmark, Problem, and Solution

Area of Science:

• Computer Vision
• Sensor Fusion
• Robotics

Background:

• RGB-Thermal (RGBT) tracking offers robustness in challenging multi-modal warranting (MMW) scenarios.
• Existing RGBT benchmarks lack representativeness due to common-scenario data, leading to failures in severe imaging conditions.
• MMW scenarios often involve invalid RGB (extreme illumination) or thermal infrared (TIR) modalities.

Purpose of the Study:

• Introduce MV-RGBT, a novel benchmark for RGBT tracking in MMW scenarios.
• Address the 'when to fuse' problem in RGBT tracking under severe imaging conditions.
• Propose MoETrack, a Mixture-of-Experts model for adaptive RGBT fusion.

Main Methods:

• Collected MV-RGBT dataset specifically from MMW scenarios with modality validity considerations.
• Divided MV-RGBT into subsets based on valid modalities for compositional evaluation.
• Developed MoETrack, employing multiple experts with confidence scoring for fusion decisions.

Main Results:

• MV-RGBT is the most diverse RGBT benchmark, featuring 36 object categories across 19 scenes.
• Demonstrated that sensor fusion is not universally beneficial in MMW scenarios.
• MoETrack achieved state-of-the-art performance on MV-RGBT, GTOT, and LasHeR benchmarks.

Conclusions:

• MV-RGBT significantly advances RGBT tracking research by focusing on challenging MMW conditions.
• Adaptive fusion strategies, like MoETrack, are crucial for robust RGBT tracking in diverse MMW scenarios.
• The findings highlight the need for careful consideration of fusion timing and necessity in RGBT tracking.