VidTr: Video Transformer Without Convolutions.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Video classification is crucial for understanding video content.
• Existing 3D convolutional networks struggle with spatio-temporal modeling efficiency.
• Transformer models show promise but often incur high memory usage.

Purpose of the Study:

• To introduce an efficient and effective video classification model.
• To address the high computational and memory costs of vanilla transformers for video.
• To improve performance on tasks requiring long-term temporal reasoning.

Main Methods:

• Developed Video Transformer (VidTr) utilizing separable-attention mechanisms.
• Implemented stacked attention layers for spatio-temporal information aggregation.
• Introduced standard deviation based topK pooling (pool_topK_std) to reduce temporal computation.

Main Results:

• VidTr achieved state-of-the-art performance on five benchmark datasets.
• Reduced memory cost by 3.3x compared to vanilla transformers with maintained performance.
• Demonstrated lower computational requirements than existing 3D networks.
• Showcased superior ability in predicting actions requiring long-term temporal reasoning.

Conclusions:

• VidTr offers a highly efficient and effective solution for video classification.
• The separable-attention and pooling strategies significantly optimize transformer performance for video.
• VidTr's architecture is particularly well-suited for complex action recognition tasks involving extended temporal dependencies.