Online Scene Semantic Understanding Based on Sparsely Correlated Network for AR.

Area of Science:

• Computer Vision
• Robotics
• Artificial Intelligence

Background:

• 2D image-based scene understanding lacks geometric detail and struggles with occlusion.
• Integrating depth sensors with geometric and semantic information presents fusion challenges.
• Real-world understanding is crucial for virtual-real mapping and interaction.

Purpose of the Study:

• To develop an online RGBD instance segmentation method that overcomes limitations of existing approaches.
• To enhance scene understanding by effectively fusing geometric and semantic information from RGBD data.
• To enable robust real-world understanding for applications like augmented reality.

Main Methods:

• Introduced a sparsely correlated network architecture (SCN) for online RGBD instance segmentation.
• Leveraged object-level RGB-D SLAM systems and frame-to-frame temporal correlations.
• Utilized sparse data generation and object layout priors to reduce complexity and improve efficiency.

Main Results:

• Achieved significantly improved accuracy and consistency in instance segmentation compared to state-of-the-art methods.
• Demonstrated real-time performance with a processing speed of 18 frames per second using sparse data.
• Validated performance on NYU Depth V2 and ScanNet V2 datasets.

Conclusions:

• Frame-to-frame correlation in video streams enhances RGBD instance segmentation accuracy and consistency.
• Sparse data processing maintains real-time performance while reducing computational complexity.
• The proposed method shows practical potential for augmented reality applications through object layout understanding.