Infra-3DRC-FusionNet: Deep Fusion of Roadside Mounted RGB Mono Camera and Three-Dimensional Automotive Radar for...

Area of Science:

• Computer Vision
• Sensor Fusion
• Artificial Intelligence

Background:

• Robust road user detection is crucial for smart infrastructure.
• Current late fusion methods struggle in adverse conditions.
• Existing deep fusion research is limited to vehicle-based setups.

Purpose of the Study:

• To propose a novel deep neural network for early fusion of camera and radar data.
• To enhance traffic user detection for roadside smart infrastructure.
• To address limitations of late fusion methods in adverse environmental conditions.

Main Methods:

• Developed a deep neural network for joint fusion of RGB camera images and 3D automotive radar point clouds.
• Utilized projected radar points to generate image anchors for guiding predictions.
• Implemented instance segmentation of radar points and post-processing for ground plane detection.

Main Results:

• Achieved precision of 92%, recall of 78%, and F1-score of 85% under various conditions.
• Demonstrated significant absolute improvements of 33% in precision, 6% in recall, and 21% in F1-score over object-level spatial fusion.
• Successfully detected road users in areas not visible to the camera.

Conclusions:

• The proposed early deep fusion methodology enhances road user detection accuracy for roadside smart infrastructure.
• This approach offers superior performance compared to traditional late fusion methods, especially in adverse conditions.
• The novel network effectively leverages complementary sensor data for robust traffic monitoring.