MMSDF: multimodal sparse dense fusion for 3D object detection.

Area of Science:

• Computer Vision
• Robotics
• Sensor Fusion

Background:

• High-precision 3D object detection in autonomous driving relies on LiDAR-camera fusion.
• Integrating geometric (LiDAR) and semantic (camera) data is challenging due to modality differences.
• Existing sparse fusion methods lack semantic richness, while dense methods are inefficient and noise-sensitive.

Purpose of the Study:

• To propose a novel multimodal sparse dense fusion (MMSDF) framework for enhanced LiDAR-camera fusion.
• To improve 3D object detection accuracy by synergistically combining sparse and dense fusion strategies.
• To address the limitations of existing fusion techniques in autonomous driving applications.

Main Methods:

• Developed a multimodal sparse dense fusion (MMSDF) framework.
• Introduced a sparse fusion attention (SFA) module for projecting LiDAR voxels to image planes and extracting semantic features.
• Implemented a dense bird's eye view (BEV) feature alignment (BFA) module using optical flow and frequency-domain convolutions.
• Designed a region of interest (roI) point-voxel fusion attention (RPVFA) module for cross-attention between point and voxel features.

Main Results:

• MMSDF achieved 88.21% accuracy on the KITTI validation set.
• MMSDF achieved 84.26% accuracy on the KITTI test set.
• Ablation studies validated the effectiveness of individual modules within the MMSDF framework.

Conclusions:

• The proposed MMSDF framework effectively integrates sparse and dense fusion strategies for superior LiDAR-camera fusion.
• MMSDF significantly enhances 3D object detection performance in autonomous driving scenarios.
• The modular design of MMSDF allows for targeted improvements and confirms the contribution of each component.