Video swin-CLSTM transformer: Enhancing human action recognition with optical flow and long-term dependencies

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Video data volume is rapidly increasing, making Human Action Recognition (HAR) crucial for applications like surveillance and sports analytics.
• Current deep learning HAR methods struggle with subtle actions, complex backgrounds, long-term dependencies, and computational efficiency.

Purpose of the Study:

• To develop an advanced HAR model that overcomes the limitations of existing deep learning approaches.
• To improve the accuracy and efficiency of recognizing human actions in videos, especially in challenging conditions.

Main Methods:

• The Video Swin-CLSTM Transformer model was developed, utilizing a Video Swin Transformer backbone.
• Optical flow information and a sparse sampling strategy were incorporated to handle background interference.
• Convolutional Long Short-Term Memory (ConvLSTM) units were embedded to capture long-term action dependencies.

Main Results:

• The model achieved 92.8% Top-1 and 99.4% Top-5 accuracy on the UCF-101 dataset, surpassing the baseline by 3.2% and 2.0%, respectively.
• Computational cost was reduced by an average of 3.3% compared to models without optical flow.
• Ablation studies confirmed the significant contributions of optical flow (2.6% accuracy increase) and ConvLSTM (1.9% accuracy increase).

Conclusions:

• The Video Swin-CLSTM Transformer demonstrates superior performance in HAR tasks compared to other Swin Transformer-based methods.
• Integrating optical flow and ConvLSTM modules effectively enhances action recognition accuracy and efficiency.
• Custom pre-training on ImageNet-1K-LSTM further boosted performance, indicating its value for HAR.