RCRFNet: Enhancing Object Detection with Self-Supervised Radar-Camera Fusion and Open-Set Recognition

Area of Science:

• Computer Vision
• Robotics
• Sensor Fusion

Background:

• Autonomous driving faces challenges in object detection due to complex environments and poor visibility.
• Millimeter-wave (mmWave) radar and visual sensors offer complementary data for improved perception.
• Existing fusion methods struggle with robustness and detecting novel or unknown objects.

Purpose of the Study:

• To develop a robust radar-camera fusion network (RCRFNet) for enhanced object detection in autonomous driving.
• To leverage self-supervised learning and open-set recognition for improved sensor data utilization.
• To address challenges posed by poor visual conditions and open-set scenarios.

Main Methods:

• Proposes the Radar-Camera Robust Fusion Network (RCRFNet).
• Employs a frustum association approach for matched radar-camera data to generate self-supervised signals.
• Integrates global and local depth consistencies with image features for object class confidence.
• Utilizes a multi-layer feature extraction backbone and multimodal detection head.

Main Results:

• RCRFNet demonstrates superior performance compared to state-of-the-art methods on the nuScenes dataset.
• Achieves robust object detection, especially in low visual visibility conditions.
• Effectively detects unknown class objects by constructing object class confidence levels.

Conclusions:

• The proposed RCRFNet significantly improves object detection robustness in challenging autonomous driving scenarios.
• Self-supervised learning and open-set recognition are effective strategies for radar-camera fusion.
• The method shows promise for real-world deployment in adverse conditions.