PE-MCAT: Leveraging Image Sensor Fusion and Adaptive Thresholds for Semi-Supervised 3D Object Detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• 3D object detection frameworks require extensive annotated data, which is costly and time-consuming to acquire from sensor data (e.g., LiDAR, cameras).
• Semi-supervised learning (SSL) offers a solution by reducing reliance on labeled data but still needs some labeled samples, which can hinder initial training of student-teacher networks.
• Existing methods struggle with insufficient local feature capture and robustness in point cloud data, especially with limited labeled samples.

Purpose of the Study:

• To propose PE-MCAT, a novel semi-supervised 3D object detection method designed to generate high-precision pseudo-labels.
• To address the limitations of insufficient labeled data in training robust student-teacher networks for 3D object detection.
• To enhance the quality of point clouds and pseudo-labels by incorporating multi-modal sensor information and advanced feature fusion techniques.

Main Methods:

• Introduced a Point Enrichment Module (PEM) that integrates image sensor data with point cloud data, employing multiple feature fusion methods to enhance local and self-features.
• Developed a multi-class adaptive threshold strategy for initial filtering and creation of a high-quality pseudo-label set from teacher network outputs.
• Implemented a joint variable threshold strategy to further refine the pseudo-label set, improving the selection of superior pseudo-labels for training.

Main Results:

• PE-MCAT demonstrated superior performance compared to state-of-the-art methods across various datasets.
• On the KITTI dataset, using only 2% of labeled samples, PE-MCAT achieved significant improvements: 0.7% mAP for cars, 3.7% for pedestrians, and 3.0% for cyclists.
• The proposed methods effectively compensated for the limitations of using minimal labeled data, enhancing the robustness and accuracy of 3D object detection.

Conclusions:

• PE-MCAT offers an effective semi-supervised approach for 3D object detection, significantly reducing the need for extensive manual annotation.
• The integration of multi-modal sensor data and advanced pseudo-labeling strategies enhances the quality and robustness of the detection models.
• The method shows strong potential for real-world sensor-based AI applications where labeled data is scarce.