Multi-Level Feature Fusion in CNN-Based Human Action Recognition: A Case Study on EfficientNet-B7

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Human action recognition is crucial for applications like healthcare and autonomous driving.
• Existing methods often rely on multiple data modalities and late fusion techniques.
• Collecting diverse data types in real-world scenarios presents challenges.

Purpose of the Study:

• To develop a multi-level fusion approach for human action recognition.
• To leverage multimodal techniques using only RGB frames as a single data source.
• To enhance model performance by combining information at early, intermediate, and late stages.

Main Methods:

• Utilized RGB frames from the NTU RGB+D dataset.
• Extracted 2D skeleton coordinates and optical flow frames from RGB data using pre-trained models.
• Implemented a multi-level fusion strategy combining information across different stages.

Main Results:

• Achieved 91.5% accuracy on the NTU RGB+D 60 dataset.
• Demonstrated significant improvements over single-modality and single-view models.
• The proposed approach showed comparable performance to state-of-the-art methods.

Conclusions:

• Multi-level fusion of features extracted from RGB frames is effective for human action recognition.
• The approach offers a practical solution by relying on a single data source.
• This method provides a robust and efficient alternative to existing techniques.