Multimodal Cross-City Semantic Segmentation Based on Similarity-Inspired Fusion and Invertible Transformation...

Area of Science:

• Computer Vision
• Remote Sensing
• Machine Learning

Background:

• Multimodal cross-city semantic segmentation adapts models from labeled source domains to unlabeled target domains in different cities.
• Domain shift is challenging due to different sensor modalities in remote sensing data.
• Traditional fusion methods neglect cross-domain information and domain shift control.

Purpose of the Study:

• To propose a similarity-inspired fusion and invertible transformation learning network (SFITNet) for multimodal cross-city semantic segmentation.
• To address challenges in feature alignment and domain shift in fused multimodal domains.
• To improve the adaptation of semantic segmentation models to new urban environments using diverse sensor data.

Main Methods:

• Developed an invertible transformation learning strategy (ITLS) using invertible neural networks (INNs) for distribution alignment.
• Designed a cross-domain similarity-inspired information interaction module (CDSiM) for effective multimodal information fusion.
• Implemented a novel network (SFITNet) integrating ITLS and CDSiM for unsupervised domain adaptation.

Main Results:

• SFITNet demonstrated superior performance compared to state-of-the-art techniques on the C2Seg-AB and Su-Wu datasets.
• The proposed ITLS effectively alleviates alignment difficulties in multimodal fused domains.
• CDSiM successfully utilizes complementary information and promotes alignment of fused domain shifts.

Conclusions:

• The proposed SFITNet effectively handles multimodal cross-city semantic segmentation challenges.
• The study highlights the importance of cross-domain similarity and invertible transformations for domain adaptation.
• SFITNet offers a promising approach for adapting remote sensing models to diverse urban environments.