MixImages: An Urban Perception AI Method Based on Polarization Multimodalities

Area of Science:

• Computer Vision
• Artificial Intelligence
• Remote Sensing

Background:

• Urban perception models often rely solely on RGB images, limiting performance in scenes with complex lighting and shadows.
• Existing methods struggle with feature confusion caused by light and shadow interplay, diminishing perception accuracy.
• Polarization data offers complementary information beyond RGB, crucial for enhancing shadow region representation.

Purpose of the Study:

• To develop a novel semantic segmentation model, MixImages, for improved urban scene perception.
• To leverage multimodal polarization data alongside RGB images to overcome limitations of unimodal approaches.
• To enhance the representation of shadow regions and improve pixel-level perception in urban environments.

Main Methods:

• Proposed a novel semantic segmentation model named MixImages.
• Integrated multimodal polarization data with traditional RGB image inputs.
• Utilized transformer architecture for its effective receptive field to capture discriminative cues.
• Conducted experiments on a dedicated polarization dataset of urban scenes.

Main Results:

• MixImages achieved a 3.43% accuracy advantage over RGB-only models in the unimodal benchmark.
• The model demonstrated a 4.29% performance improvement in the multimodal benchmark.
• Analysis of different polarization combinations provided insights for downstream task optimization.

Conclusions:

• The proposed MixImages model offers a significant advancement in urban scene perception by effectively combining RGB and polarization data.
• The integration of polarization data enhances model robustness, particularly in challenging lighting conditions like shadows.
• MixImages presents a promising new approach for pixel-level perception tasks in complex urban environments.