A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Human activity recognition (HAR) often relies on single sensor modalities like cameras or wearables, which have limitations.
• Lighting variations and sensor data scarcity hinder accurate recognition with unimodal approaches.
• Range sensors, such as LiDAR, offer robust environmental perception to complement other data sources.

Purpose of the Study:

• To propose a novel framework for human activity recognition by fusing multimodal sensor data.
• To leverage sensor fusion and multimodal machine learning to overcome limitations of single-modality systems.
• To improve the accuracy and robustness of human activity detection in diverse environments.

Main Methods:

• Utilized a framework combining RGB and point cloud data for human activity recognition.
• Employed a region-based convolutional neural network (R-CNN) for feature extraction from RGB data.
• Integrated a 3-D modified Fisher vector network to process fused sensor information.
• Developed and evaluated the model on a custom-captured multimodal dataset.

Main Results:

• Achieved a high human activity classification accuracy of 90% on the custom dataset.
• Demonstrated the effectiveness of sensor fusion in enhancing HAR performance.
• Validated the framework's capability to process and integrate data from different sensor types.

Conclusions:

• The proposed multimodal framework significantly improves human activity recognition accuracy.
• This approach offers robust perception for various applications, including sports analytics, surveillance, and autonomous vehicles.
• Sensor fusion with deep learning provides a powerful solution for complex real-world perception tasks.