Depth Super-Resolution on RGB-D Video Sequences with Large Displacement 3D Motion

Area of Science:

• Computer Vision
• Image Processing
• Machine Learning

Background:

• Video-based depth super-resolution methods enhance depth data accuracy using temporal information.
• Existing methods struggle with large displacement 3D motion, leading to compensation errors.
• These errors propagate into the fusion stage, limiting super-resolution performance.

Purpose of the Study:

• To propose a novel video-based depth super-resolution method.
• To address limitations in motion compensation and fusion stages for improved depth accuracy.
• To enhance the resolution and accuracy of depth data from RGB-D videos.

Main Methods:

• Implemented a novel motion compensation approach using 3D Nearest Neighboring Field (NNF) to handle large 3D motion displacements.
• Modeled the fusion stage as a regression problem for efficient super-resolution prediction.
• Designed a new deep convolutional neural network (CNN) architecture for the fusion process, capable of learning complex regression functions from extensive video data.

Main Results:

• The proposed 3D NNF method demonstrates superior motion compensation compared to using true motion.
• The deep CNN fusion approach effectively utilizes compensated depth images for accurate super-resolution.
• Comprehensive evaluations on various RGB-D video sequences confirm the method's superior performance.

Conclusions:

• The novel method significantly enhances depth data resolution and accuracy.
• 3D NNF is a more robust approach for motion compensation in depth super-resolution.
• The deep CNN fusion strategy effectively leverages temporal information for improved depth estimation.