DMDNet: Dual-branch multi-modal deep fusion network for V-D-T salient object detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Directly fusing multi-modal features (visible, depth, thermal) in early encoding stages introduces noise, reducing salient object detection accuracy.
• Existing methods often struggle with effectively integrating complementary information from different sensor modalities.

Purpose of the Study:

• To propose a novel dual-branch multi-modal deep fusion network (DMDNet) for improved salient object detection.
• To address the challenge of noise amplification in early feature fusion by performing fusion in the decoder phase.

Main Methods:

• DMDNet utilizes separate encoder branches for visible images and a combined branch for depth and thermal images.
• The network incorporates a modal interaction (MI) module for depth-thermal complementarity, multi-scale feature perception (MFP), and region optimization (RO) modules.
• A dual-branch fusion (DF) module integrates features bottom-to-top for final saliency map generation.

Main Results:

• DMDNet demonstrates superior performance on the VDT-2048 dataset.
• Experimental results validate the effectiveness of the proposed network architecture and fusion strategy.

Conclusions:

• The proposed DMDNet effectively reduces noise by delaying multi-modal fusion to the decoder phase.
• The network architecture successfully leverages complementary features from visible, depth, and thermal modalities for enhanced salient object detection.