Robust Human Activity Recognition by Integrating Image and Accelerometer Sensor Data Using Deep Fusion Network.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep learning for behavioral pattern recognition is gaining traction.
• Model robustness is challenged by insufficient data and changing activity types.
• Preprocessing input signals can be complex and data-intensive.

Purpose of the Study:

• To develop a generalized deep learning model for robust behavioral pattern recognition.
• To create a hybrid model that minimizes preprocessing for heterogeneous sensor data.
• To ensure consistent performance despite noise and input signal variations.

Main Methods:

• Proposed a hybrid deep learning model integrating heterogeneous sensor data (accelerometer, image).
• Utilized Convolutional Neural Network (CNN) and Convolutional Block Attention Module (CBAM) for accelerometer data.
• Employed Bidirectional Long Short-Term Memory (BiLSTM) and Residual Neural Network (ResNet) for feature extraction.

Main Results:

• Achieved 94.8% accuracy using skeleton images and accelerometer data.
• Attained 93.1% accuracy with skeleton images, coordinates, and accelerometer data.
• Demonstrated robustness with 93.4% accuracy on inverted images and 93.2% on noisy accelerometer data, showing only ~1% performance drop.

Conclusions:

• The proposed hybrid deep learning model offers robust human behavior recognition.
• The model maintains high performance across diverse and noisy sensor inputs.
• This approach minimizes preprocessing, enhancing practical applicability in real-world scenarios.