Structure-Aware Cross-Modal Transformer for Depth Completion.

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Reconstruction

Background:

• Existing depth completion methods often overlook 3D structural information in sparse depth data.
• Reliance on 2D image textures limits performance in texture-poor regions.
• Lack of explicit 3D cues hinders accurate foreground-background feature distinction.

Purpose of the Study:

• To develop a novel approach for depth completion that fully leverages inherent 3D structures.
• To enhance the representation of 2D features with 3D geometric priors.
• To improve depth completion accuracy, especially in challenging, texture-limited environments.

Main Methods:

• A Structure-aware Cross-Modal Transformer (SCMT) was proposed, utilizing a two-stream network to extract 2D and 3D features.
• Hierarchical 3D scene structures were disentangled from RGB-D input.
• Cross-modal transformers adaptively integrated multi-scale 3D structural features into the 2D feature stream.

Main Results:

• SCMT demonstrated state-of-the-art performance on benchmark datasets (KITTI, VOID, NYU).
• The method successfully incorporated 3D structural priors, improving depth boundary and object shape outline prediction.
• Enhanced 2D features led to more accurate depth completion, particularly in areas with limited texture.

Conclusions:

• The proposed SCMT effectively captures and utilizes 3D structures for superior depth completion.
• Integrating 3D information significantly overcomes limitations of texture-dependent methods.
• SCMT offers a robust solution for accurate 3D scene understanding from sparse depth data.