Balanced multi-modality knowledge mining for RGB-infrared object detection.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• RGB-Infrared object detection fuses complementary data for enhanced accuracy and robustness.
• Transformer-based methods excel at long-range dependencies but struggle with balancing modality-specific and complementary knowledge.
• Existing attention mechanisms fail to capture multi-scale relationships and both local and global information.

Purpose of the Study:

• To propose a balanced multi-modality knowledge mining method for RGB-Infrared object detection.
• To address the challenges of mining intra- and inter-modality knowledge effectively.
• To improve the capture of multi-scale object features and local/global information.

Main Methods:

• Designed a dual attention knowledge mining (DAKM) module using self-attention and cross-attention for explicit intra- and inter-modality knowledge extraction.
• Integrated multi-scale information into DAKM's attention layer to capture features across different scales and retain local/global context.
• Employed a scene-aware adaptive interaction module for focused feature fusion and a cross-layer feature refinement module to aggregate fusion layers.

Main Results:

• The proposed method demonstrated superior performance compared to existing state-of-the-art RGB-Infrared object detection techniques.
• Experiments across multiple scenes validated the effectiveness of the DAKM module and subsequent fusion strategies.
• The approach successfully balanced the mining of specific and complementary knowledge between RGB and Infrared modalities.

Conclusions:

• The developed balanced multi-modality knowledge mining method significantly advances RGB-Infrared object detection.
• The DAKM module and scene-aware fusion effectively enhance feature representation and detection performance.
• This work provides a robust solution for challenges in cross-modality fusion for object detection.