Learning Complementary Correlations for Depth Super-Resolution With Incomplete Data in Real World

Area of Science:

• Computer Vision
• 3D Sensing
• Machine Learning

Background:

• Depth information is crucial for visual perception but often captured incompletely and at low resolution by sensors like time-of-flight (ToF) cameras.
• This limitation leads to degraded visual perception quality.

Purpose of the Study:

• To address the task of depth super-resolution (DSR) using incomplete data.
• To recover dense, high-resolution depth maps from incomplete, low-resolution inputs.

Main Methods:

• Introduction of a novel Incomplete DSR (IDSR) framework with two branches: one for DSR and another for depth completion (DC).
• Proposal of two key modules: Joint Correlation Learning (JCL) to learn inter-branch relationships and Iterative-Cross (IC) for fusing higher-level representations.
• The framework enhances the learning of complementary information between the DSR and DC branches.

Main Results:

• The proposed IDSR framework demonstrates effectiveness in recovering high-frequency details and filling missing pixels.
• Achieved state-of-the-art performance on both real-world RGB-D-D and synthetic NYUv2 datasets.
• The JCL and IC modules significantly enhance the fusion of complementary information for precise depth map prediction.

Conclusions:

• The novel IDSR framework successfully tackles the challenge of depth super-resolution with incomplete data.
• The proposed joint learning approach with JCL and IC modules offers a significant advancement in depth map recovery.
• The method provides a robust solution for improving the quality of depth perception from sensor data.