Nonlinear Spiking Neural Systems for Thermal Image Semantic Segmentation Networks.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Biomedical Imaging

Background:

• RGB and thermal images offer complementary information, crucial for low-light conditions.
• Spatial discrepancies in RGB-Thermal data hinder multimodal feature fusion in semantic segmentation.
• Existing methods struggle with information loss during fusion, limiting performance.

Purpose of the Study:

• To propose a novel network, CSPM-SNPNet, for effective RGB-Thermal semantic segmentation.
• To address challenges in multimodal feature fusion and spatial information loss.
• To enhance feature extraction and restore spatial context in segmented images.

Main Methods:

• Developed a channel-space fusion module for integrating RGB and thermal image features.
• Introduced a nonlinear spiking neural P system with convolution (ConvSNP) for enhanced decoding.
• Utilized the proposed CSPM-SNPNet model for RGB-Thermal semantic segmentation tasks.

Main Results:

• CSPM-SNPNet demonstrated significant improvements in segmentation performance on MFNet and PST900 datasets.
• Achieved a 0.5% mIOU increase on MFNet and 1.8% on PST900 compared to existing methods.
• Effectively restored spatial contextual information, enhancing feature representation.

Conclusions:

• The proposed CSPM-SNPNet effectively overcomes spatial discrepancies in RGB-Thermal data.
• The novel fusion module and spiking neural P system enhance multimodal feature integration and extraction.
• CSPM-SNPNet shows superior performance, particularly in complex and challenging visual scenes.