Action recognition using attention-based spatio-temporal VLAD networks and adaptive video sequences optimization.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Human action recognition faces challenges in effectively capturing video-level spatio-temporal features.
• Convolutional Neural Networks (CNNs) struggle to model long-range temporal information crucial for complex actions.

Purpose of the Study:

• To develop a novel attention-based spatio-temporal VLAD network (AST-VLAD) for improved human action recognition.
• To propose an automatic approach for adaptive video sequences optimization (AVSO) to enhance action-related frame representation.

Main Methods:

• The AST-VLAD network aggregates informative deep features using a self-attention model.
• Adaptive Video Sequences Optimization (AVSO) employs shot segmentation and dynamic weighted sampling to prioritize action frames.
• Self-attention models intrinsic spatio-temporal relationships, moving beyond simple pooling methods.

Main Results:

• The proposed AST-VLAD with AVSO demonstrates superior or comparable performance on public benchmarks.
• Achieved classification accuracy of 73.1% on HMDB51 and 96.0% on UCF101 datasets.
• The method effectively models long-range temporal dependencies and optimizes video sequences for recognition.

Conclusions:

• The AST-VLAD network combined with AVSO offers a robust solution for human action recognition.
• This approach effectively addresses the limitations of CNNs in modeling long-range temporal dynamics.
• The proposed method enhances the accuracy and efficiency of video-based action classification.