An Effective Video Transformer With Synchronized Spatiotemporal and Spatial Self-Attention for Action Recognition

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Convolutional neural networks (CNNs) have dominated vision tasks, but vision Transformers (ViTs) now lead in image recognition.
• Video Transformers lag behind image Transformers in research attention and efficiency due to parameter and training differences.

Purpose of the Study:

• To propose and validate techniques for improving video understanding using Transformer architectures.
• To enhance the efficiency and effectiveness of video Transformers.

Main Methods:

• Introduced a synchronized spatiotemporal and spatial attention (SSTSA) scheme for better spatiotemporal feature representation.
• Developed a motion spotlighting module to incorporate short-term motion into RGB inputs for a single-stream Transformer.
• Implemented an intraclass frame interlacing method as a video augmentation technique.

Main Results:

• The proposed video Transformer, incorporating SSTSA, motion spotlighting, and frame interlacing, demonstrated superior performance.
• Achieved state-of-the-art results on the Kinetics400 and Something-Something-v2 datasets.

Conclusions:

• The developed techniques significantly advance video understanding capabilities of Transformer models.
• The proposed methods offer a more efficient and effective approach to video Transformer design.