Event-Driven Video Restoration With Spiking-Convolutional Architecture.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Neuroscience-inspired Computing

Background:

• Event cameras offer high temporal resolution and dynamic range, advancing low-level vision tasks.
• Existing event-based methods often fail to fully leverage spatial and temporal event information due to CNN limitations.
• Insufficient use of spatial distribution and temporal relations in neighboring events hinders video restoration quality.

Purpose of the Study:

• To propose a novel Spiking-Convolutional Network (SC-Net) for improved event-driven video restoration.
• To enhance the extraction of temporal correlations within event data and spatial consistency between events and frames.
• To accurately restore detailed textures and sharp edges in low-quality video sequences.

Main Methods:

• Developed a hybrid Spiking Neural Network (SNN) and Convolutional Neural Network (CNN) architecture (SC-Net).
• SNN component captures temporal correlations in sparse event data.
• CNN component transforms sparse events into brightness priors for detailed texture awareness.

Main Results:

• The SC-Net effectively utilizes both temporal event correlations and spatial information from events and frames.
• Demonstrated superior performance in deblurring, super-resolution, and deraining tasks compared to existing methods.
• Validated effectiveness on both synthetic and real-world benchmark datasets.

Conclusions:

• The proposed SC-Net architecture significantly improves event-driven video restoration.
• The hybrid SNN-CNN approach successfully addresses limitations of previous event-based methods.
• SC-Net offers a promising direction for enhancing video quality using event camera data.