Multi-scale full spike pattern for semantic segmentation.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Neuroscience

Background:

• Spiking neural networks (SNNs) offer low-power, brain-inspired computation but face challenges in semantic segmentation tasks.
• Existing SNN models for semantic segmentation exhibit poor performance and high memory overhead, limiting their practical application.
• Deep direct training of SNNs with surrogate gradients is an emerging area with potential for improved efficiency.

Purpose of the Study:

• To develop an efficient SNN-based model for pixel-level semantic segmentation.
• To address the performance limitations and high memory/energy overhead of current SNN segmentation models.
• To demonstrate the effectiveness of deep SNNs in computer vision tasks requiring precise spatial understanding.

Main Methods:

• Proposed the Multi-scale and Full Spike Segmentation network (MFS-Seg), a deep SNN trained with surrogate gradients.
• Introduced an Efficient Fully-Spike Residual block (EFS-Res) using depthwise separable convolution to handle spiking noise and improve feature representation.
• Theoretically analyzed EFS-Res using block dynamical isometry theory to prevent performance degradation.

Main Results:

• MFS-Seg achieved comparable semantic segmentation performance to the mainstream UNet architecture on benchmark datasets (Camvid, DDD17, DSEC-Semantic).
• The model demonstrated significant reductions in memory overhead (up to 31x fewer parameters) and power consumption (over 13x reduction).
• Visualizations confirmed MFS-Seg's ability to effectively extract object edge features.

Conclusions:

• MFS-Seg represents a significant advancement in applying deep SNNs to semantic segmentation tasks.
• The proposed EFS-Res block effectively mitigates SNN-specific challenges, enhancing performance and efficiency.
• The findings underscore the potential of SNNs as a viable, low-power alternative for complex computer vision applications.