Learning Effective RGB-D Representations for Scene Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• RGB scene recognition is advanced using deep convolutional networks (CNNs) and large datasets.
• RGB-Depth (RGB-D) scene recognition lags due to depth data limitations for training deep learning models.
• Existing depth sensors have a limited range, failing to capture distant scene objects.

Purpose of the Study:

• To address limitations in RGB-D scene recognition, specifically the lack of sufficient depth data and the short range of depth sensors.
• To propose a novel architecture and training approach for effective depth-specific feature learning.
• To introduce a new dataset and model for RGB-D video scene recognition.

Main Methods:

• A two-step training approach with weak supervision via patches to learn depth-specific features directly.
• Development of an architecture combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs) for video analysis.
• Introduction of the ISIA RGB-D video dataset for evaluating RGB-D scene recognition using videos.

Main Results:

• The proposed method achieves state-of-the-art performance on RGB-D image datasets (NYUD2, SUN RGB-D).
• The approach demonstrates superior results in RGB-D video scene recognition using the new ISIA RGB-D dataset.
• The model effectively learns complementary multimodal features and overcomes depth sensor range limitations by utilizing video data.

Conclusions:

• The developed architecture and training strategy effectively learn depth-specific features for RGB-D scene recognition.
• Utilizing RGB-D videos significantly enhances scene recognition by capturing comprehensive depth information over time.
• The ISIA RGB-D video dataset provides a valuable resource for advancing research in video-based RGB-D scene recognition.