A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds.

Area of Science:

• Computer Vision
• Autonomous Driving Systems
• Sensor Fusion

Background:

• Bridging the modality gap between camera images and LiDAR point clouds is crucial for autonomous driving.
• Current fusion methods struggle with effective cross-modal feature integration.
• Semantic segmentation performance is limited by the inability to fully leverage multi-sensor data.

Purpose of the Study:

• To propose a novel Cross-Modal Fusion (CMF) framework for enhanced multi-sensor data integration.
• To achieve state-of-the-art performance in semantic segmentation tasks for autonomous driving.
• To address the limitations of existing fusion methods in handling cross-modal features.

Main Methods:

• Projecting LiDAR point clouds onto camera coordinates using perspective projection for spatio-depth information.
• Employing a two-stream feature extraction network for separate modality processing.
• Implementing a residual fusion module (RCF) with cross-modal attention for multilevel fusion.
• Designing a perceptual alignment loss integrating cross-entropy and feature matching terms.

Main Results:

• Achieved state-of-the-art mean intersection over union (mIoU) scores of 64.2% on SemanticKITTI and 79.3% on nuScenes.
• Demonstrated superior accuracy and enhanced robustness in complex driving scenarios compared to existing methods.
• Ablation studies confirmed the effectiveness of cross-modal attention and perceptually guided cross-entropy loss (Pgce) in improving segmentation.

Conclusions:

• The CMF framework successfully bridges the modality gap between camera and LiDAR data.
• The proposed attention-driven architecture and perceptual alignment loss significantly enhance semantic segmentation performance.
• CMF offers a robust and accurate solution for multi-sensor fusion in autonomous driving systems.