Cross-modal transformer fusion via local sampling for drone RGB-infrared object detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Remote Sensing

Background:

• Visible and infrared feature fusion is crucial for drone-based RGB-IR object detection.
• Existing transformer-based methods often use downsampled features, losing critical local details and reducing accuracy, especially for small or complex objects.

Purpose of the Study:

• To propose a novel Cross-Modal Transformer Fusion via Local Sampling (CTFLS) network for improved drone-based object detection.
• To address the limitations of low-resolution feature fusion in current transformer models.

Main Methods:

• A two-stream strategy is employed to capture intra-modal and inter-modal information from high-resolution feature maps.
• A Local Cascade Transformer (LCT) module with local intra-modal and cross-modal transformer blocks extracts rich information from high-resolution features.
• A Detail-Enhanced Mixed-Convolution Attention (DMA) module enhances fused feature representation for subtle textures and global context.

Main Results:

• The proposed CTFLS network effectively captures both intra-modal and inter-modal information from high-resolution feature maps.
• Local fusion via cross-attention on sampled local features explores deeper complementary relationships between modalities.
• The DMA module improves the representation of fused features, capturing subtle textures and global context.

Conclusions:

• The CTFLS network outperforms state-of-the-art methods in drone-based object detection.
• The method demonstrates significant improvements in detecting small objects and distinguishing objects with similar structures.