Geometry-aware multimodal fusion for large-scale 3D scene understanding.

Area of Science:

• Computer Vision
• Robotics
• Geospatial Data Analysis

Background:

• Multimodal 3D semantic segmentation is vital for engineering but challenged by complex geometry, scale variations, and sparse LiDAR data.
• Existing methods struggle with real-world scene complexity and data sparsity, limiting applications in autonomous systems and 3D visualization.

Purpose of the Study:

• To develop a robust and efficient multimodal 3D understanding method addressing challenges in LiDAR-based semantic segmentation.
• To enhance semantic discrimination in sparse 3D regions by fusing LiDAR and image data.

Main Methods:

• A dual-path 3D feature extractor with position-based encoding for spatial structures.
• A geometry-aware adaptive aggregation module using learnable kernels and hybrid weighting.
• Fusion of LiDAR point clouds with passive optical image features.

Main Results:

• Achieved real-time inference (25ms/sample) with a lightweight framework (5.2M parameters).
• Demonstrated competitive performance on Semantic3D, SemanticKITTI (71.2% mIoU), and nuScenes, outperforming RandLA-Net by 15.3% on SemanticKITTI.
• Validated effectiveness on real-world colored point clouds for outdoor environment semantic structuring.

Conclusions:

• The proposed adaptive geometry fusion method offers robust and efficient multimodal 3D understanding.
• The lightweight and real-time capabilities enable deployment in resource-constrained sensing and visualization platforms.
• Successfully provides semantically structured 3D representations for downstream applications like 3D display.