Recurrent Spatial-Temporal Attention Network for Action Recognition in Videos.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Recurrent Neural Networks (RNNs) are popular for video action recognition.
• Traditional RNNs struggle with high-dimensional video data and complex human dynamics.
• Capturing intricate action information across various motion scales remains a challenge.

Purpose of the Study:

• To propose a novel Recurrent Spatial-Temporal Attention Network (RSTAN) for improved video action recognition.
• To address the limitations of traditional RNNs in handling complex video data.
• To enhance the ability of RNNs to capture detailed human dynamics and motion scales.

Main Methods:

• Introduced a spatial-temporal attention mechanism to adaptively identify key features within the global video context for each RNN time-step prediction.
• Reinforced Long Short-Term Memory (LSTM) with a novel spatial-temporal attention module for learning compact, relevant action representations.
• Designed an attention-driven appearance-motion fusion strategy for jointly training appearance and motion LSTMs in an end-to-end framework.
• Developed actor-attention regularization to guide the attention mechanism towards important actor regions.

Main Results:

• The proposed RSTAN demonstrated superior performance compared to other recent RNN-based approaches on the UCF101 and HMDB51 datasets.
• RSTAN achieved state-of-the-art results on the JHMDB dataset.
• Experimental validation confirmed the effectiveness of the spatial-temporal attention mechanism and fusion strategy.

Conclusions:

• RSTAN effectively addresses the challenges of high dimensionality and complex dynamics in video action recognition.
• The novel attention mechanisms and fusion strategy significantly improve the accuracy of action recognition.
• The proposed method offers a robust and efficient solution for advanced video understanding tasks.