NeRF-Det++: Incorporating Semantic Cues and Perspective-Aware Depth Supervision for Indoor Multi-View 3D Detection.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Neural Radiance Fields (NeRF) have advanced multi-view 3D detection.
• Existing NeRF-based detectors face challenges in semantic ambiguity, sampling, and depth supervision.

Purpose of the Study:

• To address the limitations of NeRF-Det for improved multi-view 3D object detection.
• To introduce novel techniques for enhanced semantic understanding and geometric clue utilization.

Main Methods:

• Semantic Enhancement: Projecting 3D segmentation onto 2D semantic maps for supervision.
• Perspective-Aware Sampling: Densely sampling near the camera and sparsely at a distance.
• Ordinal Residual Depth Supervision: Combining depth bin classification and residual regression.

Main Results:

• NeRF-Det++ demonstrates superior performance on ScanNetV2 and ARKITScenes datasets.
• Achieved +1.9% mAP @0.25 and +3.5% mAP @0.50 improvements over NeRF-Det on ScanNetV2.
• The proposed methods effectively enhance semantic awareness and depth learning.

Conclusions:

• NeRF-Det++ significantly improves multi-view 3D detection by tackling key shortcomings.
• The novel techniques offer a more robust and accurate approach to 3D scene understanding.
• The open-source code facilitates further research and development in the field.