STA-TSN: Spatial-Temporal Attention Temporal Segment Network for action recognition in video

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Deep learning action recognition models often struggle with complex, multi-stage actions due to a focus on short-term motions.
• Existing Temporal Segment Networks (TSN) capture long-term information but are susceptible to interference from irrelevant video frames and areas.

Purpose of the Study:

• To propose a novel Spatial-Temporal Attention Temporal Segment Networks (STA-TSN) model that addresses the limitations of current action recognition techniques.
• To enhance the ability of action recognition models to adaptively focus on salient spatial and temporal features within videos.

Main Methods:

• Introduced a soft attention mechanism into TSN to create STA-TSN, enabling adaptive focus on key features.
• Developed a multi-scale spatial focus feature enhancement strategy using spatial pyramid pooling and soft attention.
• Designed a key frame exploration module employing a Long-Short Term Memory (LSTM) based soft attention mechanism for temporal attention weighting.
• Implemented a temporal-attention regularization to guide key frame exploration.

Main Results:

• The proposed STA-TSN model demonstrated superior performance compared to the standard TSN.
• STA-TSN achieved state-of-the-art results on four public datasets: UCF101, HMDB51, JHMDB, and THUMOS14.
• The model effectively mitigates interference from irrelevant frames and areas by focusing on critical spatio-temporal information.

Conclusions:

• STA-TSN significantly advances the field of video-based action recognition, particularly for complex actions.
• The integration of spatial and temporal attention mechanisms provides a robust solution for adaptive feature focusing.
• The proposed method offers a promising direction for developing more accurate and efficient deep learning models for action recognition.