MDM: Modality decoupling for visible and infrared Mamba-based object detection

Area of Science:

• Computer Vision
• Machine Learning
• Sensor Fusion

Background:

• Visible and infrared (IR) sensor fusion significantly improves object detection accuracy and robustness.
• Existing fusion methods struggle with multi-scale targets, significant modality differences, and high computational complexity.
• Current CNNs have static kernels, while Transformers exhibit quadratic complexity, limiting practical applications.

Purpose of the Study:

• To develop a versatile object detection model adaptable to both conventional and remote sensing scenarios.
• To address the limitations of existing fusion techniques, particularly concerning multi-scale targets and modality differences.
• To reduce computational overhead in cross-modal fusion for long sequences.

Main Methods:

• Designed Global and Local Mamba modules for extracting global context and local receptive fields.
• Introduced a Modality Decouple module to separate modality-agnostic and modality-specific features.
• Implemented differentiated fusion strategies using lightweight Spatial Attention, Spatial Mamba, and Channel Mamba modules, alongside a novel cross-modal interaction paradigm for Mamba.

Main Results:

• The proposed Modality Decouple Module (MDM) achieves state-of-the-art (SOTA) performance efficiently across multiple datasets.
• Demonstrated superior performance on conventional scene object detection (FLIR, LLVIP, M3FD) and remote sensing small object detection (VEDAI, DroneVehicle).
• The method effectively handles multi-scale targets and exhibits strong robustness, addressing challenges in small object detection.

Conclusions:

• The MDM significantly enhances object detection accuracy and robustness through efficient multimodal fusion.
• This study pioneers the exploration of Mamba's interaction paradigm for cross-modal fusion, unlocking its potential.
• The proposed approach offers an efficient and robust solution for diverse object detection tasks, outperforming existing methods.