Learn to Enhance Sparse Spike Streams.

Area of Science:

• Computer Vision
• Deep Learning
• Biologically Inspired Computing

Background:

• High-speed vision tasks are challenging due to limitations in traditional cameras.
• Spike cameras offer high temporal resolution but struggle with sparse data in low-light conditions.
• Existing spike-based methods degrade in performance under low-light, high-speed scenarios.

Purpose of the Study:

• To introduce SS2DS, the first deep learning framework for enhancing sparse spike streams into dense ones.
• To address the performance degradation of spike-based vision systems in low-light, high-speed environments.
• To improve the effectiveness of spike cameras for challenging computer vision tasks.

Main Methods:

• SS2DS estimates spike firing frequency from sparse spike streams.
• A neural network enhances the estimated spike firing frequency.
• The enhanced frequency sequence is decoded into a dense spike stream.
• Both synthetic and real-world datasets from a third-generation spike camera were used for evaluation.

Main Results:

• Enhanced spike streams showed significant improvements: +0.78 MA, -18.42 BRISQUE, and -1.42 NIQE compared to sparse streams.
• Downstream tasks like 3D reconstruction benefited, with +1.325 dB PSNR and +0.005 SSIM.
• Super-resolution tasks also improved, showing +0.63 MA, -13.67 BRISQUE, and -1.28 NIQE.

Conclusions:

• SS2DS effectively enhances sparse spike streams, improving temporal distribution and visual quality.
• The framework mitigates performance degradation in low-light, high-speed vision tasks.
• The developed method shows broad applicability, benefiting various spike-based vision applications.