High Speed and High Dynamic Range Video with an Event Camera.

Area of Science:

• Computer Vision
• Robotics
• Sensor Technology

Background:

• Event cameras offer high temporal resolution, high dynamic range, and no motion blur, surpassing conventional cameras.
• Reconstructing intensity images from event streams is challenging due to the ill-posed nature of the problem and reliance on hand-crafted priors in existing methods.

Purpose of the Study:

• To develop a data-driven approach for reconstructing intensity images from event streams, bypassing hand-crafted priors.
• To create a novel recurrent neural network capable of synthesizing high-quality videos from event data.
• To extend the reconstruction method for color images and demonstrate its effectiveness for downstream computer vision tasks.

Main Methods:

• A novel recurrent neural network architecture was designed for event stream to video reconstruction.
• The network was trained on simulated event data using a perceptual loss function to ensure natural image statistics.
• The approach was extended to synthesize color images from color event streams.

Main Results:

• The proposed network significantly outperforms state-of-the-art methods in image quality and operates in real-time.
• High-framerate videos of high-speed phenomena and high dynamic range reconstructions in challenging conditions were successfully synthesized.
• Reconstructed event data proved effective as an intermediate representation for computer vision tasks like object classification and visual-inertial odometry.

Conclusions:

• Data-driven learning, using recurrent networks and perceptual loss, effectively addresses the challenge of intensity image reconstruction from event streams.
• The developed method provides superior image quality and real-time performance, enabling new applications for event cameras.
• The reconstructions serve as a versatile intermediate representation, enhancing the performance of standard computer vision algorithms on event data.