SPADnet: deep RGB-SPAD sensor fusion assisted by monocular depth estimation

Area of Science:

• Computer Vision
• Photonics
• Machine Learning

Background:

• Single-photon light detection and ranging (LiDAR) utilizes single-photon detectors (SPADs) for advanced 3D imaging.
• Estimating scene depth from noisy SPAD measurements presents significant challenges.

Purpose of the Study:

• To develop a robust deep sensor fusion strategy for accurate depth estimation.
• To introduce SPADnet, a neural network architecture for combining corrupted SPAD data and 2D images.

Main Methods:

• Proposed SPADnet architecture integrating monocular depth estimation with SPAD denoising and sensor fusion.
• Employed advanced network training techniques for RGB-SPAD fusion.
• Validated the approach using both simulated and captured data.

Main Results:

• Achieved state-of-the-art results in RGB-SPAD fusion for depth estimation.
• Demonstrated superior performance compared to existing fusion networks.
• SPADnet exhibits enhanced computational efficiency.

Conclusions:

• The proposed deep sensor fusion strategy effectively addresses challenges in SPAD data processing.
• SPADnet offers a computationally efficient and high-performing solution for 3D depth estimation.
• This work advances LiDAR capabilities through improved sensor fusion techniques.