Enhanced Geometry and Semantics for Camera-Based 3D Semantic Scene Completion

Area of Science:

• Computer Vision
• 3D Scene Understanding
• Artificial Intelligence

Background:

• Vision-centric Semantic Scene Completion (SSC) is vital for machine decision-making and planning.
• Current SSC methods struggle with depth errors and ambiguities during 2D-to-3D transformations.
• Rich visual cues and low cost make vision-centric SSC a popular paradigm.

Purpose of the Study:

• To enhance the accuracy and robustness of Semantic Scene Completion (SSC).
• To address limitations in depth prediction and feature representation in 2D-to-3D scene understanding.
• To improve geometric prediction and semantic reasoning for complex 3D scenes.

Main Methods:

• Developed an Optical Flow-Guided (OFG) Depth-Net leveraging pre-trained models and optical flow for improved depth accuracy.
• Introduced a depth ambiguity-mitigated feature lifting strategy using deformable cross-attention in 3D pixel space.
• Customized residual voxel and sparse UNet subnetworks for enhanced geometric prediction and multi-scale semantic reasoning.

Main Results:

• Achieved significant performance improvements over state-of-the-art methods on SemanticKITTI, SSCBench-KITTI-360, and Occ3D-nuScene benchmarks.
• Demonstrated enhanced depth prediction accuracy, particularly in regions with significant depth changes.
• Showcased improved geometric prediction and consistent semantic reasoning across various scales.

Conclusions:

• The proposed method effectively overcomes limitations of existing SSC approaches by mitigating depth errors and ambiguities.
• The integration of optical flow guidance and advanced feature lifting strategies leads to superior 3D scene understanding.
• This work advances the field of 3D perception, paving the way for more capable autonomous systems.