A multidimensional feature fusion network based on MGSE and TAAC for video-based human action recognition

Area of Science:

• Computer Vision and Artificial Intelligence
• Machine Learning for Human Action Recognition (HAR)

Background:

• Current video-based HAR methods struggle with abstract feature extraction.
• Limited action recognition capabilities exist for special populations like prisoners and the elderly.
• Existing HAR technologies lack comprehensive feature extraction for complex human actions.

Purpose of the Study:

• To propose a novel multidimensional feature fusion network, P-MTSC3D, for enhanced human action recognition.
• To address limitations in abstract feature extraction and recognition for special personnel.
• To improve the accuracy and applicability of HAR in diverse real-world scenarios.

Main Methods:

• Developed a parallel network (P-MTSC3D) integrating context modeling and temporal adaptive attention.
• Employed a three-branch architecture: basic feature extraction, spatial-channel feature extraction (MGSE modules), and temporal feature extraction (TAAC units).
• Validated the network on UCF101 and HMDB51 datasets, and introduced a new Prison Action (PA) dataset.

Main Results:

• Achieved high recognition accuracies: 97.92% on UCF101 and 75.59% on HMDB51.
• Demonstrated superior performance compared to state-of-the-art HAR networks.
• Showcased effectiveness in specialized scenarios through experiments on the newly constructed PA dataset.

Conclusions:

• The P-MTSC3D network effectively extracts multidimensional features for accurate human action recognition.
• The proposed method offers significant improvements over existing HAR techniques, especially for complex actions and specific populations.
• P-MTSC3D shows strong potential for real-world applications, including surveillance and monitoring of special personnel.