Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Open-vocabulary 3D Object Detection (OV-3DDet) is challenging due to the need to detect objects from arbitrary novel categories in 3D scenes.
• Existing methods struggle with accurately localizing and classifying objects outside of predefined base categories.

Purpose of the Study:

• To propose CoDAv2, a unified framework for open-vocabulary 3D object detection.
• To improve both the localization and classification of novel 3D objects, especially with limited base categories.

Main Methods:

• 3D Novel Object Discovery (3D-NOD) strategy uses 3D geometries and 2D semantic priors to generate pseudo labels for novel objects.
• 3D-NODE enhances 3D-NOD with an Enrichment strategy to improve novel object distribution and localization.
• Discovery-driven Cross-modal Alignment (DCMA) module aligns 3D point cloud, 2D, and text features for classification, refined iteratively.
• Box-DCMA incorporates 2D box guidance to enhance classification accuracy against background noise.

Main Results:

• CoDAv2 significantly outperforms existing methods in novel object detection on SUN-RGBD and ScanNetv2 datasets.
• Achieved AP_Novel of 9.17 on SUN-RGBD (vs. 3.61) and 9.12 on ScanNetv2 (vs. 3.74).
• Demonstrates superior performance in localizing and classifying a wider range of novel 3D objects.

Conclusions:

• CoDAv2 provides an effective unified framework for open-vocabulary 3D object detection.
• The proposed 3D-NODE and DCMA modules are key innovations driving the performance gains.
• The framework shows strong potential for real-world applications requiring detection of diverse, unseen objects.