Depth Privileged Scene Recognition via Dual Attention Hallucination

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• RGB-D scene recognition leverages depth data for enhanced performance.
• Depth sensor inaccessibility limits practical applications of RGB-D recognition.
• The depth-privileged setting assumes depth data is available only during training.

Purpose of the Study:

• To develop a method for RGB-D scene recognition in a depth-privileged setting.
• To hallucinate depth attention from RGB images for improved scene recognition.
• To overcome the limitations of depth sensor unavailability during testing.

Main Methods:

• A novel model based on modulated deformable convolutional layers is proposed.
• Dual attention mechanisms (importance weight and spatial transformation) are hallucinated from RGB data.
• A weighted loss function is designed to mitigate negative transfer based on depth attention quality.

Main Results:

• The proposed method significantly outperforms existing state-of-the-art approaches in depth-privileged scene recognition.
• Experiments conducted on SUN RGB-D and NYUDv2 benchmarks validate the effectiveness of the approach.
• The hallucinated depth attention effectively complements RGB information.

Conclusions:

• The developed technique successfully addresses the challenge of depth unavailability during testing in RGB-D scene recognition.
• Hallucinating depth attention from RGB data is a viable strategy for improving recognition performance.
• The method offers a practical solution for deploying RGB-D recognition systems in real-world scenarios with limited depth sensing capabilities.