Unified 3D and 4D Panoptic Segmentation via Dynamic Shifting Networks

Area of Science:

• Computer Vision
• Robotics
• Autonomous Driving

Background:

• Autonomous driving systems require robust 3D perception.
• Existing methods often focus on either object detection or semantic segmentation, not a unified scene parsing.
• LiDAR data presents unique challenges due to complex point cloud distributions.

Purpose of the Study:

• To develop a unified framework for LiDAR-based Panoptic Segmentation.
• To address the challenge of parsing both objects and scenes in 3D point clouds.
• To extend the framework for 4D Panoptic Segmentation, incorporating temporal information for consistent instance prediction across frames.

Main Methods:

• Proposed Dynamic Shifting Network (DS-Net) with a dynamic shifting module for complex point clouds.
• Introduced an efficient, learnable clustering module that adapts kernel functions.
• Developed 4D-DS-Net by constructing 4D data volumes from aligned LiDAR scans for unified temporal clustering.

Main Results:

• DS-Net demonstrated effectiveness as a panoptic segmentation framework for point clouds.
• 4D-DS-Net achieved superior performance in 4D Panoptic Segmentation by unifying temporal instance clustering.
• Experiments on SemanticKITTI and Panoptic nuScenes datasets validated the proposed methods.

Conclusions:

• The proposed DS-Net and 4D-DS-Net offer a significant advancement in holistic 3D and 4D perception for autonomous driving.
• The dynamic shifting module effectively handles complex LiDAR data distributions.
• Unified temporal processing in 4D-DS-Net ensures consistent instance identification across multiple frames.