Plug-and-play algorithm for 3D guided video super-resolution of single-photon LiDAR data

Area of Science:

• Computational Imaging
• Photonics and Sensors
• Computer Vision

Background:

• Single-photon avalanche diodes (SPADs) offer picosecond timing resolution for photon detection, crucial for applications like LiDAR and low-light imaging.
• SPAD cameras capture high-speed binary images, enabling 3D reconstruction of dynamic environments, but face challenges with motion blur and lower spatial resolution compared to passive cameras.

Purpose of the Study:

• To develop a novel computational imaging algorithm for improved 3D reconstruction of moving scenes using SPAD data.
• To address motion blur and enhance the spatial resolution of depth images derived from high-speed SPAD events.
• To generate non-blurred, high-resolution depth images synchronized with the frame rate of conventional passive cameras.

Main Methods:

• A plug-and-play approach within an optimization framework was employed.
• The algorithm alternates between guided video super-resolution of the 3D scene and precise image realignment using optical flow.
• The method processes high-speed SPAD events to produce depth images at a lower, synchronized frame rate.

Main Results:

• Experiments on synthetic data demonstrated significant improvements in image resolution across various signal-to-noise ratios and photon levels.
• Validation with real-world SPAD measurements confirmed the method's effectiveness in dynamic scenarios, including fast-moving objects and human subjects.
• Successful application across different ranges (3m to 325m) and sensor types (consumer-grade to short-wave infrared SPAD cameras) was achieved.

Conclusions:

• The proposed computational imaging algorithm effectively mitigates motion blur and enhances spatial resolution in SPAD-based 3D reconstruction.
• The method proves robust and versatile for reconstructing dynamic scenes under diverse real-world conditions.
• This advancement holds significant potential for applications requiring high-fidelity 3D information from SPAD sensors in challenging environments.