MGAF: LiDAR-Camera 3D Object Detection With Multiple Guidance and Adaptive Fusion

Area of Science:

• Computer Vision
• Robotics
• Autonomous Driving

Background:

• 3D object detection methods using Bird's-Eye-View (BEV) are advancing.
• Existing methods often neglect the synergistic potential between LiDAR and camera data.

Purpose of the Study:

• To propose a novel multi-modality 3D object detection framework, MGAF, that leverages complementary LiDAR-camera interactions.
• To enhance feature representation and fusion for improved 3D detection accuracy.

Main Methods:

• Introduced sparse depth guidance (SDG) and LiDAR occupancy guidance (LOG) for rich 3D feature generation.
• Developed an Adaptive Fusion Dual Transformer (AFDT) for enhanced global and bidirectional BEV feature interaction.
• Incorporated multi-scale dual-path transformers (MSDPT) and a temporal fusion module for expanded receptive fields and temporal aggregation.

Main Results:

• The proposed MGAF framework achieved state-of-the-art performance on nuScenes, Waymo Open Dataset, and Argoverse2.
• The Adaptive Fusion Dual Transformer (AFDT) demonstrated generalizability and superior performance when applied to other models.
• The method effectively improved 3D object detection by enhancing LiDAR-camera feature fusion and temporal consistency.

Conclusions:

• MGAF offers a significant advancement in multi-modality 3D object detection by effectively fusing LiDAR and camera information.
• The proposed adaptive fusion and guidance mechanisms are crucial for robust performance in complex driving scenarios.
• The framework's generalizability highlights its potential for broader applications in autonomous systems.