Self-Supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration

Area of Science:

• Computer Vision
• Robotics
• Machine Learning

Background:

• Autonomous driving systems require accurate 3D perception.
• Integrating data from multiple sensors like cameras and LiDAR is crucial.
• Self-supervised learning offers a promising approach to reduce reliance on labeled data.

Purpose of the Study:

• Introduce NCLR, a novel self-supervised framework for 3D perception.
• Develop a 2D-3D neural calibration pretext task for camera-LiDAR alignment.
• Enhance the understanding of LiDAR-to-camera extrinsic parameters.

Main Methods:

• Propose learnable transformation alignment to bridge domain gaps between image and point cloud features.
• Identify overlapping regions using fused features for robust matching.
• Establish dense 2D-3D correspondences to estimate rigid poses for sensor calibration.

Main Results:

• NCLR achieves effective alignment of image and point cloud data at both fine-grained and holistic levels.
• Pre-trained NCLR backbone significantly improves performance on downstream tasks like 3D semantic segmentation, object detection, and panoptic segmentation.
• Demonstrates superiority over existing self-supervised methods on various datasets.

Conclusions:

• Joint learning from multi-modal data enhances network understanding and representation effectiveness.
• NCLR provides a robust and effective self-supervised approach for 3D perception in autonomous driving.
• The proposed method advances the field of sensor fusion and calibration for autonomous systems.