Spatiotemporal Interaction Residual Networks with Pseudo3D for Video Action Recognition

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Action recognition is a key challenge in computer vision.
• Two-stream Convolutional Neural Networks (CNNs) and 3D CNNs are current leading methods.

Purpose of the Study:

• To develop a novel deep network that integrates spatial and temporal learning for improved action recognition.
• Introduce the spatiotemporal interaction residual network with pseudo3D (STINP).

Main Methods:

• Proposed a novel STINP architecture using two branches based on residual networks (ResNets).
• Integrated pseudo3D blocks into the spatial branch for appearance and inter-frame interaction learning.
• Employed a multiplication operation to fuse spatial and temporal branches for interactive learning.

Main Results:

• The STINP model demonstrated superior performance on the UCF101 and HMDB51 action recognition datasets.
• Achieved better video recognition results compared to existing state-of-the-art algorithms.

Conclusions:

• The proposed STINP effectively combines spatial and temporal information for robust action recognition.
• STINP offers a promising advancement in deep learning for video analysis.