Spike-EIFNet: Lightweight Spike-Driven Event-Image Fusion Network for Accurate and Efficient Semantic Segmentation

Area of Science:

• Robotics and Artificial Intelligence
• Computer Vision
• Neuromorphic Engineering

Background:

• Semantic segmentation is vital for intelligent robotics, but RGB camera performance degrades in low-light or fast-motion scenarios.
• Event cameras offer high temporal resolution and low latency, excelling in challenging conditions.
• Existing event-image fusion methods often use simple strategies, leading to suboptimal accuracy and high energy costs.

Purpose of the Study:

• To develop a lightweight and energy-efficient event-image fusion network for robust semantic segmentation.
• To leverage the strengths of spiking neural networks (SNNs) for multimodal fusion in robotics.
• To improve accuracy and reduce computational demands compared to traditional methods.

Main Methods:

• Proposed Spike-EIFNet, a dual-branch SNN encoder for parallel event and image processing.
• Introduced a spike-driven cross-modal fusion (SCMF) module with modality-aware fine-grained extraction (MFE) and cross-modal interaction and fusion (CIF).
• Incorporated a lightweight feature enhancement (LFE) module for refined feature representation and fusion.

Main Results:

• Achieved 67.34% mIoU on DDD17 and 58.09% mIoU on DSEC-Semantic datasets.
• Demonstrated significant energy savings: 72.83x on DDD17 and 100.26x on DSEC-Semantic.
• Outperformed ANN-based methods in energy efficiency and SNN-based methods in accuracy-efficiency tradeoff.

Conclusions:

• Spike-EIFNet offers a highly efficient and accurate solution for semantic segmentation using event-image fusion.
• The proposed SNN-based approach provides a favorable balance between accuracy and energy consumption for intelligent systems.
• This work advances the application of neuromorphic computing in real-world robotic perception tasks.