Sensing Diversity and Sparsity Models for Event Generation and Video Reconstruction from Events.

Area of Science:

• Event-based vision
• Computer vision
• Deep learning

Background:

• Event-to-video (E2V) reconstruction and video-to-events (V2E) simulation are key areas in event-based vision.
• Existing deep learning models for E2V are complex and hard to interpret.
• Current event simulators focus on realism but lack research into improving generation processes.

Purpose of the Study:

• To propose a lightweight, interpretable model-based deep network for E2V reconstruction.
• To explore diverse event generation strategies for V2E simulation.
• To validate improved video reconstruction using a video-to-events-to-video (V2E2V) architecture.

Main Methods:

• Developed a convolutional ISTA network (CISTA) using sparse representation and algorithm unfolding for E2V.
• Incorporated Long Short-Term Temporal Consistency (LSTC) constraints for enhanced temporal coherence.
• Introduced interleaved pixels with varying contrast thresholds and bandwidth for V2E generation.

Main Results:

• The proposed CISTA-LSTC network outperforms state-of-the-art methods in E2V reconstruction.
• Achieved superior temporal consistency in reconstructed videos.
• Sensing diversity in event generation revealed finer details, leading to improved reconstruction quality.

Conclusions:

• The proposed CISTA-LSTC model offers an effective and interpretable solution for E2V reconstruction.
• Novel V2E generation strategies enhance detail extraction and overall video reconstruction.
• The V2E2V framework successfully validates the benefits of improved event generation.