Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation.

Area of Science:

• Computer Vision
• Deep Learning
• 3D Semantic Segmentation

Background:

• 3D spatial information enhances semantic segmentation.
• Current methods use separate streams for RGB and 3D data, increasing inference time.
• This limits real-time applications.

Purpose of the Study:

• To propose an efficient method for integrating 3D spatial information into RGB feature learning for semantic segmentation.
• To develop a novel convolutional module that leverages 3D geometry.
• To achieve real-time performance without compromising accuracy.

Main Methods:

• Introduced Spatial information guided Convolution (S-Conv) for direct integration of 3D spatial data.
• S-Conv infers kernel sampling offsets guided by 3D information.
• Developed Spatial information Guided convolutional Network (SGNet) using S-Conv.

Main Results:

• S-Conv enhances geometric perception without significant parameter or computational cost increases.
• SGNet achieves real-time inference speeds.
• State-of-the-art performance was demonstrated on the NYUDv2 and SUNRGBD datasets.

Conclusions:

• S-Conv offers an efficient approach for integrating 3D spatial information in semantic segmentation.
• SGNet provides a viable solution for real-time 3D semantic segmentation tasks.
• The proposed method effectively leverages geometric information for improved performance.