Towards Interpretable Camera and LiDAR Data Fusion for Autonomous Ground Vehicles Localisation

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Deep learning methods show promise for ego-motion estimation, outperforming traditional geometric approaches.
• Existing research is limited by a lack of multimodal datasets, often focusing on single-sensor data.
• Accurate ego-motion estimation is crucial for autonomous navigation and robotics.

Purpose of the Study:

• To address the need for multimodal data in ego-motion estimation.
• To propose and evaluate an end-to-end deep learning architecture for sensor fusion in odometry.
• To introduce the LboroAV2 multimodal dataset for autonomous vehicle research.

Main Methods:

• Collected a novel multimodal dataset (LboroAV2) with camera, LiDAR, ultrasound, e-compass, and rotary encoder.
• Developed an end-to-end deep learning architecture combining a convolutional encoder and a recurrent neural network.
• Fused RGB images and LiDAR data for ego-motion estimation within the proposed network.

Main Results:

• The proposed deep learning architecture successfully fuses multimodal sensor data for odometry.
• The convolutional encoder generates a compressed representation aiding visualization and sequential information transfer.
• Experiments on LboroAV2 and KITTI datasets demonstrate superior performance compared to existing methods.

Conclusions:

• The developed deep learning approach offers a significant advancement in ego-motion estimation through multimodal sensor fusion.
• The LboroAV2 dataset provides a valuable resource for future research in autonomous vehicle perception.
• The open-source release of the code facilitates reproducibility and further development in the field.