RCRFNet: Enhancing Object Detection with Self-Supervised Radar-Camera Fusion and Open-Set Recognition

Minwei Chen ¹, Yajun Liu ¹, Zenghui Zhang ¹

⁰Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, Shanghai 200240, China.

Sensors (basel, Switzerland) +

|

August 10, 2024

View abstract on PubMed

Summary

This study introduces a radar-camera fusion network (RCRFNet) for robust object detection in autonomous driving. The network excels in poor visibility and detects unknown objects by combining millimeter-wave radar and visual data.

Area Of Science

Computer Vision
Robotics
Sensor Fusion

Background

Autonomous driving faces challenges in object detection due to complex environments and poor visibility.
Millimeter-wave (mmWave) radar and visual sensors offer complementary data for improved perception.
Existing fusion methods struggle with robustness and detecting novel or unknown objects.

Purpose Of The Study

To develop a robust radar-camera fusion network (RCRFNet) for enhanced object detection in autonomous driving.
To leverage self-supervised learning and open-set recognition for improved sensor data utilization.
To address challenges posed by poor visual conditions and open-set scenarios.

Main Methods

Proposes the Radar-Camera Robust Fusion Network (RCRFNet).
Employs a frustum association approach for matched radar-camera data to generate self-supervised signals.
Integrates global and local depth consistencies with image features for object class confidence.
Utilizes a multi-layer feature extraction backbone and multimodal detection head.

Main Results

RCRFNet demonstrates superior performance compared to state-of-the-art methods on the nuScenes dataset.
Achieves robust object detection, especially in low visual visibility conditions.
Effectively detects unknown class objects by constructing object class confidence levels.

Conclusions

The proposed RCRFNet significantly improves object detection robustness in challenging autonomous driving scenarios.
Self-supervised learning and open-set recognition are effective strategies for radar-camera fusion.
The method shows promise for real-world deployment in adverse conditions.

Keywords:

autonomous driving open-set recognition radar–camera fusion self-supervised learning target detection