MAF-Net: A multimodal data fusion approach for human action recognition

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• 3D skeleton-based human activity recognition is robust but struggles with spatiotemporal dynamics and multimodal integration.
• Existing methods face challenges in effectively combining skeletal and RGB data for comprehensive analysis.

Purpose of the Study:

• To propose a novel multimodal fusion framework for enhanced 3D skeleton-based human activity recognition.
• To address limitations in capturing spatiotemporal dynamics and integrating diverse data modalities.

Main Methods:

• Developed a multimodal fusion framework utilizing optical flow-based key frame extraction and data augmentation.
• Employed self-attention and skeletal attention modules for fusing skeletal and RGB streams.
• Implemented a late fusion strategy to combine skeletal and RGB features, capturing spatial and temporal dependencies.

Main Results:

• Achieved superior performance on benchmark datasets (NTU RGB+D, SYSU, UTD-MHAD) compared to existing models.
• Demonstrated improved accuracy in human activity recognition through effective multimodal feature integration.
• Validated the framework's robustness and effectiveness in capturing complex spatiotemporal information.

Conclusions:

• The proposed multimodal fusion framework significantly enhances 3D skeleton-based human activity recognition accuracy.
• The method offers a robust foundation for future multimodal integration in real-time applications.
• Potential applications include surveillance, healthcare, and human-computer interaction.