CrossModalSync: joint temporal-spatial fusion for semantic scene segmentation in large-scale scenes

Area of Science:

• Computer Vision
• Robotics
• Artificial Intelligence

Background:

• Point cloud semantic segmentation is crucial for autonomous vehicles navigating complex environments.
• Existing methods struggle with cumulative errors and information loss in dynamic scenes.

Purpose of the Study:

• To develop a framework balancing accuracy and efficiency for point cloud semantic segmentation.
• To address challenges like cumulative errors and the "many-to-one" mapping problem.

Main Methods:

• Utilizing temporal alignment (TA) and projection multi-scale convolution (PMC) to capture inter-frame correlations and local details.
• Employing priority point retention (PPR) to preserve critical 3D information and mitigate the "many-to-one" mapping issue.
• Integrating LiDAR and camera data through multimodal fusion for enhanced perception.

Main Results:

• Achieved state-of-the-art performance on SemanticKITTI and nuScenes datasets.
• Demonstrated improved accuracy and computational efficiency compared to existing methods.
• Successfully detected occluded objects and dynamic entities in complex scenes.

Conclusions:

• The proposed framework offers a robust solution for point cloud semantic segmentation in autonomous driving.
• The combination of TA, PMC, PPR, and multimodal fusion significantly enhances segmentation performance.
• The method shows great potential for real-world applications in autonomous systems.