PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes.

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Manual 2D annotation for self-driving car perception systems is labor-intensive.
• Existing datasets lack annotations for rare viewpoints, limiting model generalization.

Purpose of the Study:

• To develop a novel approach for generating high-quality panoptic labels and images from any viewpoint.
• To enhance the generalization ability of perception models for autonomous vehicles.

Main Methods:

• PanopticNeRF-360 combines coarse 3D annotations with noisy 2D semantic cues.
• It leverages 3D bounding primitives and 2D predictions to optimize geometry and semantics.
• A hybrid scene representation using MLP and hash grids enhances appearance and semantics.

Main Results:

• PanopticNeRF-360 achieves state-of-the-art performance on the KITTI-360 dataset.
• The method successfully generates high-fidelity, multi-view, and consistent labels and images.
• It demonstrates improved geometry and semantic fusion, effectively filtering annotation noise.

Conclusions:

• PanopticNeRF-360 offers a powerful solution for creating comprehensive 3D scene understanding for autonomous driving.
• The approach effectively addresses limitations in existing datasets and annotation methods.
• It enables omnidirectional rendering of detailed and consistent scene representations.