A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

Area of Science:

• Computer Vision
• Sensor Fusion
• Autonomous Driving Systems

Background:

• Current pedestrian detection methods often rely on RGB and lidar, which struggle in scattered environments and do not mimic human visual perception.
• Radar offers a complementary sensing modality to overcome limitations of lidar and vision in challenging conditions.

Purpose of the Study:

• To investigate the feasibility of fusing lidar, radar, and RGB data for enhanced pedestrian detection.
• To develop a multimodal sensor fusion architecture for autonomous driving applications.
• To evaluate the performance of a semantic segmentation-based approach for detecting pedestrians.

Main Methods:

• A fully connected convolutional neural network architecture based on SegNet was employed for pixel-wise semantic segmentation.
• Lidar and radar data were transformed into 2D grayscale images, while RGB images were used with three channels.
• A novel extrinsic calibration matrix method using singular value decomposition was proposed for sensor alignment.

Main Results:

• The proposed fusion method achieved high performance metrics, including a training mean pixel accuracy of 99.7% and training mean intersection over union (IoU) of 99.5%.
• Testing yielded a mean IoU of 94.4% and a testing pixel accuracy of 96.2%, demonstrating effectiveness despite a small custom dataset.
• The model showed good performance in detecting pedestrians, even with some observed overfitting.

Conclusions:

• The fusion of lidar, radar, and RGB data using semantic segmentation is a feasible and effective approach for pedestrian detection.
• This multimodal strategy offers advantages in detecting pedestrians with less ambiguity, similar to human visual perception.
• Further research with larger datasets is recommended to optimize training and mitigate overfitting for robust real-world deployment.