Learning 3D Representation From Auto-Labeled 2D Object Boxes.

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• LiDAR representation learning with limited annotations shows promise.
• Existing methods use superpixels for cross-modal contrastive learning, causing semantic ambiguity in 3D features.
• This ambiguity impairs the performance of 3D object detection and segmentation.

Purpose of the Study:

• To leverage unlabeled LiDAR-camera pairs for a novel pre-training pipeline.
• To learn directly from category space and group 3D features of the same object.
• To improve 3D network performance under limited annotations.

Main Methods:

• Autolabeled 2D object boxes are generated using an open-vocabulary object detector.
• A box-to-label-maps algorithm creates pixel-wise label maps from 2D boxes.
• A dual-space pre-training 3D network utilizes pseudo labels for category recognition and object segmentation.
• AdaptPro module explores unpaired 3D features using category prototypes for enhanced fine-tuning.

Main Results:

• The proposed method achieves state-of-the-art performance on the nuScenes benchmark.
• The method also demonstrates state-of-the-art results on the SemanticKITTI benchmark.
• The approach effectively reduces semantic ambiguity and improves 3D feature representation.

Conclusions:

• The novel pre-training pipeline effectively addresses limitations in current LiDAR representation learning.
• The method demonstrates superior performance in 3D object detection and segmentation tasks.
• The approach offers a promising direction for leveraging unlabeled multi-modal data in autonomous driving.