TwinsTNet: Broad-View Twins Transformer Network for Bi-Modal Salient Object Detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Bi-modal salient object detection (BSOD) requires effectively integrating complementary information from different data sources like RGB and thermal/depth.
• Existing BSOD models face challenges in handling the distinct information distributions and achieving robust fusion due to limitations in convolutional operations and attention mechanisms.

Purpose of the Study:

• To develop a novel network architecture, TwinsTNet, capable of accurately performing BSOD by addressing the limitations of existing methods.
• To enhance the fusion of complementary information across different modalities and improve long-range dependency modeling.

Main Methods:

• Proposed a uniform broad-view Twins Transformer Network (TwinsTNet) for BSOD.
• Introduced Cross-Modal Federated Attention (CMFA) for efficient bi-modal information fusion via element-wise global dependency.
• Developed Semantic Consistency Attention Loss to supervise co-attention features and Cross-Scale Retracing Attention (CSRA) for flexible inter-layer interactions.

Main Results:

• TwinsTNet demonstrated superior performance compared to twenty-two state-of-the-art BSOD models.
• The proposed CMFA and CSRA mechanisms effectively mitigated inductive bias in modality and layer dimensions, enhancing the network's representational capability.
• The model achieved state-of-the-art results across ten benchmark BSOD datasets.

Conclusions:

• TwinsTNet offers a significant advancement in bi-modal salient object detection by effectively fusing multi-modal data.
• The novel attention mechanisms (CMFA and CSRA) provide a robust solution for challenges in cross-modal information fusion and inter-layer interaction.
• The proposed method establishes a new state-of-the-art in BSOD performance.