A Multimodal Fusion Approach for Human Activity Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Human Activity Recognition (HAR) is crucial for various applications, requiring analysis of human motion and behavior from sensor data.
• Existing methods often rely on single data modalities, limiting comprehensive understanding of complex activities.

Purpose of the Study:

• To propose a novel multimodal approach for video-based Human Activity Recognition (HAR).
• To leverage 3D visual data (RGB + depth) for enhanced HAR by creating diverse 2D representations.

Main Methods:

• Utilized 3D skeletal sequences and RGB data from an RGB+depth camera.
• Transformed data into six distinct 2D image representations, including spectral and dynamic images.
• Employed six Convolutional Neural Networks (CNNs) for feature extraction, followed by feature fusion and Support Vector Machine (SVM) classification.

Main Results:

• The proposed multimodal approach demonstrated superior performance in human activity recognition tasks.
• Experiments included single-view, cross-view, and cross-subject evaluations on a challenging dataset.
• Outperformed three other state-of-the-art methods in most experimental settings.

Conclusions:

• The multimodal fusion of diverse 2D representations derived from 3D data significantly enhances video-based HAR.
• The approach offers a robust and effective solution for recognizing human activities in complex scenarios.
• This work contributes to advancing the field of HAR through innovative data representation and fusion techniques.