Optical flow estimation from event-based cameras and spiking neural networks.

Area of Science:

• Computer Vision
• Neuromorphic Engineering
• Artificial Intelligence

Background:

• Event-based cameras offer low power, low latency, and high dynamic range, ideal for challenging applications.
• Spiking Neural Networks (SNNs) coupled with neuromorphic hardware enable real-time, low-power systems.

Purpose of the Study:

• To develop a system for dense optical flow estimation in driving scenarios using event-based camera data and SNNs.
• To create an efficient and accurate optical flow estimation model tailored for real-time applications.

Main Methods:

• A U-Net-like Spiking Neural Network (SNN) architecture was designed and trained using the DSEC dataset.
• Supervised training with back-propagation and a surrogate gradient was employed, optimizing for minimal error norm and angle.
• 3D convolutions and separable convolutions were utilized to capture temporal dynamics and create a lightweight model.

Main Results:

• The proposed SNN successfully performed dense optical flow estimations for driving scenarios.
• The model demonstrated reasonably accurate results while maintaining a lightweight architecture compared to existing methods.
• The use of 3D convolutions effectively captured the dynamic nature of the event-based data.

Conclusions:

• The developed SNN system effectively estimates optical flow from event-based camera data, showing promise for real-time applications.
• The combination of event sensors, SNNs, and efficient network design offers a viable solution for low-power, high-performance computer vision tasks.