High-Resolution Depth Maps Imaging via Attention-Based Hierarchical Multi-Modal Fusion

Area of Science:

• Computer Vision
• Image Processing

Background:

• Depth maps are crucial for many applications but often have low spatial resolution from consumer RGB-D cameras.
• Guided depth map super-resolution (DSR) uses high-resolution (HR) RGB images to enhance low-resolution (LR) depth maps.
• A key challenge in DSR is effectively handling structural consistency and inconsistencies between modalities.

Purpose of the Study:

• To propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided depth map super-resolution.
• To improve the accuracy and efficiency of reconstructing high-resolution depth maps from low-resolution inputs.

Main Methods:

• Developed a multi-modal attention-based fusion (MMAF) strategy for hierarchical convolutional layers.
• Incorporated a feature enhancement block for valuable feature selection and a feature recalibration block for modality similarity unification.
• Introduced a bi-directional hierarchical feature collaboration (BHFC) module to integrate multi-scale spatial and structural information.

Main Results:

• The proposed AHMF network demonstrates superior performance compared to state-of-the-art methods.
• Achieved significant improvements in reconstruction accuracy.
• Showcased enhanced running speed and memory efficiency.

Conclusions:

• The novel AHMF network effectively addresses the challenges in guided DSR.
• The proposed fusion and collaboration modules enable better utilization of multi-modal information.
• The method offers a promising solution for high-quality depth map reconstruction.