Mitigating Modality Discrepancies for RGB-T Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Image Processing

Background:

• Semantic segmentation models benefit from visible and thermal infrared (RGB-T) data for illumination robustness.
• Existing RGB-T models often use basic fusion methods that ignore modality discrepancies, limiting cross-modal information exploitation.

Purpose of the Study:

• To propose MDRNet+, an improved network for RGB-T semantic segmentation.
• To address the limitations of primitive fusion strategies in current RGB-T models.
• To enhance the exploitation of complementary information between visible and thermal infrared images.

Main Methods:

• Introduced a 'bridging-then-fusing' strategy to mitigate modality discrepancies before feature fusion.
• Designed an improved Modality Discrepancy Reduction (MDR+) subnetwork for unimodal feature processing.
• Utilized channel-weighted fusion (CWF) modules and multiscale context modules (MSC, MCC) for adaptive feature integration and context capture.
• Developed the RTSS dataset for urban scene understanding to address data scarcity.

Main Results:

• The proposed MDRNet+ model demonstrated superior performance compared to state-of-the-art methods.
• Experiments on MFNet, PST900, and the new RTSS datasets confirmed the model's effectiveness.
• The 'bridging-then-fusing' strategy successfully reduced modality discrepancies, leading to better feature fusion.

Conclusions:

• MDRNet+ offers a significant advancement in RGB-T semantic segmentation.
• The novel fusion strategy and context modules effectively leverage complementary RGB-T information.
• The developed RTSS dataset provides a valuable resource for advancing urban scene understanding research.