Adjusting modal contributions via differential-driven SoftMax fusion for RGB-IR object detection

Visible light (RGB) and infrared (IR) fusion enhances object detection in complex environments by leveraging complementary modalities. However, adaptively balancing modal contributions under extreme distribution imbalance remains challenging, while transformer-based methods incur heavy computational costs. To address this, we propose Diff-Fusion, which uses semantically prepared inter-modal differences to generate spatially varying complementary weights for modal contributions in efficient RGB-IR detection. To improve modality-specific feature encoding, we apply lightweight RGB and IR detail enhancement adapters to the pretrained encoder. Then, we design the adaptive differential-driven feature fusion (ADFF) module to learn the relative contribution weights of RGB and IR features during fusion. To prepare multi-modal features for contrast-driven fusion, ADFF first learns shared channel-wise semantic weights to construct comparable spatial response maps. Finally, ADFF derives paired contribution weights from the inter-modal difference through a parameter-free complementary gate, enabling context-aware feature complementarity. The design combines modality-specific adaptation with spatially varying, normalized relative weighting in a parameter-efficient fusion framework. Extensive experiments on the FLIR, LLVIP, and M³FD datasets demonstrate that Diff-Fusion achieves leading reported mAP values of 50.6%, 71.4%, and 54.0%, respectively, with compact trainable adaptation and fusion modules. These results support the effectiveness of its compact adaptation and spatially adaptive fusion design for RGB-IR object detection.