Cross-Modal Collaboration and Robust Feature Classifier for Open-Vocabulary 3D Object Detection.

Area of Science:

• Robotics and Autonomous Systems
• Computer Vision
• Artificial Intelligence

Background:

• Multi-sensor fusion (LiDAR, camera) is crucial for 3D object detection in autonomous driving.
• Existing models fail to detect unknown or novel objects, limiting real-world application.
• Open-vocabulary 3D object detection is essential for handling environmental complexity.

Purpose of the Study:

• To address the challenges of localizing and classifying novel objects in 3D environments.
• To improve the accuracy and robustness of 3D object detection for unknown categories.
• To advance the capabilities of autonomous systems in complex, dynamic settings.

Main Methods:

• Proposed Cross-modal Collaboration for iterative refinement of object localization between LiDAR and camera data.
• Developed Robust Feature Classifier to map semantic vectors and cluster visual features for accurate novel object classification.
• Integrated 2D image regions of interest with 3D point cloud precision for enhanced detection.

Main Results:

• Achieved state-of-the-art performance in open-vocabulary 3D object detection across diverse scenarios.
• Demonstrated significant improvements in localization accuracy for novel objects.
• Showcased enhanced classification robustness, preventing misidentification of unknown objects.

Conclusions:

• The proposed Cross-modal Collaboration and Robust Feature Classifier effectively handle novel object detection.
• This research significantly advances the field of open-vocabulary 3D object detection.
• The method offers a robust solution for autonomous systems requiring detection of a wide range of objects.