Decouple-Then-Synergize: A Self-Paced Collaborative Learning Network for RGB-T Snowy Urban Scene Parsing.

Area of Science:

• Computer Vision
• Remote Sensing
• Artificial Intelligence

Background:

• Urban scene analysis relies on fusing RGB and thermal infrared images, but performance degrades in snowy conditions.
• Existing enhancement methods increase model complexity and can cause interference.
• A new framework is needed to effectively fuse multimodal data under adverse weather.

Purpose of the Study:

• To develop a novel framework for robust RGB and thermal image fusion in urban scene analysis, particularly under snowy conditions.
• To decouple the fusion task into frequency-oriented enhancement and spatial semantic fusion.
• To improve model performance and reduce complexity compared to existing methods.

Main Methods:

• Proposed a "decouple-then-synergize" framework comprising FRENet (frequency restoration enhancement network) and SIFNet (spatial interactive fusion network).
• FRENet employs an asymmetric strategy for RGB color and thermal target enhancement with spectral refinement.
• SIFNet uses a Mamba zipper fusion module for semantic interaction and a reconstruction task for feature integration.

Main Results:

• The proposed framework, FRENet-CL and SIFNet-CL, demonstrated superior performance on SUS and PST900 datasets.
• Achieved state-of-the-art results in urban scene parsing, outperforming existing methods.
• The self-paced curriculum facilitated effective collaboration and knowledge exchange between networks.

Conclusions:

• The "decouple-then-synergize" framework effectively addresses performance degradation in snowy conditions for RGB and thermal image fusion.
• The proposed method enhances urban scene analysis accuracy and robustness.
• This work provides a significant advancement in multimodal image fusion for challenging environments.