A Hybrid Network for Large-Scale Action Recognition from RGB and Depth Modalities

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Large-scale action recognition presents significant challenges for current methods.
• Existing state-of-the-art approaches do not fully address these complexities.

Purpose of the Study:

• To propose a novel hybrid network for multi-modal, large-scale action recognition.
• To leverage the strengths of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) architectures.

Main Methods:

• Developed a hybrid network integrating CNN and RNN components.
• Utilized proposed weighted dynamic images for feature extraction.
• Fused features using canonical correlation analysis.
• Classified actions using a linear Support Vector Machine (SVM).

Main Results:

• Achieved state-of-the-art performance on ChaLearn LAP IsoGD, NTU RGB+D, and M2I datasets.
• Outperformed existing methods by over 10 percentage points in certain cases.
• Demonstrated the effectiveness of the proposed weighted dynamic images and hybrid network architecture.

Conclusions:

• The proposed hybrid network effectively addresses challenges in large-scale action recognition.
• The integration of CNN and RNN with weighted dynamic images offers a significant advancement.
• The method shows strong potential for real-world multi-modal action recognition applications.