RepAttn3D: Re-parameterizing 3D attention with spatiotemporal augmentation for video understanding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Structural re-parameterization is common in image tasks using CNNs and MLPs.
• Integrating re-parameterization with attention mechanisms in video analysis is underexplored.
• Video analysis faces high computational costs, especially during inference.

Purpose of the Study:

• To investigate re-parameterization of 3D attention mechanisms for video understanding.
• To incorporate a spatiotemporal coherence prior to enhance video feature learning.
• To address computational challenges in video analysis tasks.

Main Methods:

• Proposing a SpatioTemporally Augmented 3D Attention (STA-3DA) module for Transformer architectures.
• Integrating 3D, spatial, and temporal attention branches during training.
• Merging attention branches into a single 3D operation with learned weights during testing.

Main Results:

• The STA-3DA module learns more robust video features with negligible inference overhead.
• The proposed module effectively replaces standard 3D attention in Transformer models, improving performance.
• Achieved competitive video understanding performance on Kinetics-400 and Something-Something V2 datasets.

Conclusions:

• The STA-3DA module offers an efficient and effective approach to enhance video understanding.
• Re-parameterization of 3D attention with spatiotemporal priors is a promising direction for video analysis.
• The method provides a practical solution for reducing computational costs in video Transformer models.