Semantic Guidance Fusion Network for Cross-Modal Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Multimodal data fusion is crucial for advanced segmentation tasks.
• Integrating diverse sensor data like depth and thermal imaging presents challenges in cross-modal feature extraction.
• Existing methods struggle with effectively amalgamating unique characteristics from different modalities.

Purpose of the Study:

• To introduce an innovative cross-modal fusion network, the Semantic Guidance Fusion Network (SGFN).
• To enhance bi-modal feature extraction and effectively integrate diverse modalities for segmentation.
• To address the challenge of merging disparate data traits within a unified framework.

Main Methods:

• Developed the Semantic Guidance Fusion Network (SGFN) for multimodal segmentation.
• Introduced a Semantic Guidance Module (SGM) to boost feature extraction.
• Utilized a learnable Semantic Guidance Convolution (SGC) to merge intensity and gradient data from different modalities.

Main Results:

• The SGFN demonstrated superior performance and generalization across multiple benchmark datasets (NYU Depth V2, SUN-RGBD, Cityscapes, MFNet, ZJU).
• Achieved performance comparable to leading models like CMXNET on the DELIVER dataset using a bi-modal SGFN.
• Validated the effectiveness of the SGM and SGC in improving cross-modal fusion.

Conclusions:

• The SGFN offers a robust and effective solution for multimodal segmentation.
• The proposed semantic guidance approach significantly enhances the integration of diverse data modalities.
• SGFN represents a significant advancement in cross-modal fusion for computer vision tasks.