Self-Supervised Learning for RGB-Guided Depth Enhancement by Exploiting the Dependency Between RGB and Depth.

Area of Science:

• Computer Vision
• Image Processing
• Machine Learning

Background:

• Time-of-flight (ToF) sensors produce depth images with significant noise and degradation.
• Existing RGB-guided depth enhancement methods struggle with realistic noise and the lack of clean training data.

Purpose of the Study:

• To develop a self-supervised learning method for RGB-guided depth image enhancement.
• To address the limitations of supervised methods by eliminating the need for noisy-clean depth image pairs.

Main Methods:

• Exploiting the inherent dependency between RGB and depth images for self-supervision.
• Maximizing cross-modal dependency between RGB and depth to guide enhancement.
• Augmenting the formulation with optimal transport theory for improved performance.

Main Results:

• The proposed method significantly outperforms state-of-the-art techniques in depth denoising, multi-path interference suppression, and hole filling.
• Demonstrated remarkable superiority on real-world data, effectively handling complex and realistic degradations.
• Achieved superior enhancement performance compared to existing methods on both synthetic and real-world datasets.

Conclusions:

• Self-supervised learning based on cross-modal dependency is effective for RGB-guided depth image enhancement.
• The method offers a robust solution for improving the quality of depth images from ToF sensors.
• This approach provides a practical alternative for depth enhancement when clean data is unavailable.