RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion

Area of Science:

• Computer Vision
• Machine Learning
• 3D Reconstruction

Background:

• Indoor depth images often have missing values due to sensor limitations and environmental factors.
• Incomplete depth maps hinder subsequent computer vision tasks, necessitating effective depth completion methods.
• Existing techniques struggle with large, contiguous missing regions common in indoor environments.

Purpose of the Study:

• To develop a novel network for accurate depth completion from incomplete RGB-D data in indoor settings.
• To address the challenge of extensive missing depth values in raw depth images.
• To improve the performance of depth completion, especially in realistic indoor scenarios.

Main Methods:

• A two-branch end-to-end fusion network, RDFC-GAN, was designed, accepting RGB and incomplete depth images.
• The first branch uses an encoder-decoder structure with Manhattan world assumption and normal maps for local depth regression.
• The second branch employs an RGB-depth fusion CycleGAN for detailed depth map generation, with adaptive fusion modules (W-AdaIN) and pseudo depth map training.

Main Results:

• The RDFC-GAN method demonstrated significant enhancements in depth completion performance.
• The network excels particularly in realistic indoor environments with extensive missing depth data.
• Evaluations on NYU-Depth V2 and SUN RGB-D datasets validated the proposed approach.

Conclusions:

• RDFC-GAN effectively addresses the challenge of depth completion in indoor scenes with significant missing data.
• The novel two-branch fusion network architecture provides superior performance compared to existing methods.
• This work contributes a robust solution for generating dense and accurate depth maps from incomplete inputs.