PLGS: Robust Panoptic Lifting With 3D Gaussian Splatting.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Neural Radiance Fields (NeRF) offer panoptic lifting but suffer from slow training and rendering.
• 3D Gaussian Splatting (3DGS) provides faster alternatives but lacks inherent smoothness, making it vulnerable to noisy mask supervision.
• Existing methods struggle with efficient and robust panoptic segmentation, especially when dealing with imperfect input data.

Purpose of the Study:

• To develop an efficient 3D Gaussian Splatting (3DGS) method for robust panoptic segmentation from noisy 2D masks.
• To introduce smoothness into 3DGS for improved handling of mask supervision.
• To enhance the speed and accuracy of panoptic lifting compared to NeRF-based approaches.

Main Methods:

• Proposed PLGS, a panoptic-aware structured 3D Gaussian model to ensure smoothness and reduce noise sensitivity.
• Introduced semantic anchor points for reliable initialization and smooth regularization of 3D Gaussians.
• Implemented a self-training approach with pseudo-labels and projected 2D instance masks for cross-view consistent supervision.

Main Results:

• PLGS demonstrates superior efficiency and segmentation quality compared to existing NeRF-based methods.
• The method effectively handles noisy 2D segmentation masks, producing consistent panoptic outputs.
• Achieved state-of-the-art performance on various benchmarks for panoptic segmentation tasks.

Conclusions:

• PLGS successfully integrates 3D Gaussian Splatting for efficient and robust panoptic segmentation.
• The proposed techniques address the limitations of conventional 3DGS in handling noisy supervision.
• This work offers a promising direction for real-time, high-quality 3D scene understanding.