DFormer++: Improving RGBD Representation Learning for Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• RGB-D semantic segmentation faces challenges due to the mismatch between RGB-pretrained models and RGB-D data.
• Existing methods often fail to effectively encode 3D geometric relationships present in depth maps.

Purpose of the Study:

• To propose DFormer++, a novel pretrain-and-finetune framework to learn transferable representations for RGB-D semantic segmentation.
• To address the common mismatch problem in RGB-D semantic segmentation.

Main Methods:

• Developed DFormer++, a framework that pretrains backbones using image-depth pairs from ImageNet-1K, enabling direct encoding of RGB-D representations.
• Introduced RGB-D attention blocks with a novel attention mechanism tailored for encoding both RGB and depth information.

Main Results:

• DFormer++ effectively avoids mismatched encoding of 3D geometry by RGB-pretrained backbones.
• The tailored architecture reduces redundant parameters, achieving efficient and accurate RGB-D perception.
• Achieved state-of-the-art performance on three popular RGB-D semantic segmentation benchmarks.

Conclusions:

• The proposed DFormer++ framework successfully learns robust RGB-D representations.
• The novel architecture and pretraining strategy significantly improve performance and efficiency in RGB-D semantic segmentation.