Gate-Shift-Fuse for Video Action Recognition.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) excel at image recognition but 3D CNNs struggle with video action recognition due to high computational costs.
• Existing 3D CNN complexity reduction methods use fixed, hand-designed approaches.
• Large-scale annotated datasets are often required for training complex 3D CNNs effectively.

Purpose of the Study:

• To introduce Gate-Shift-Fuse (GSF), a novel module for efficient spatio-temporal feature extraction in video analysis.
• To enable existing 2D CNNs to perform high-performance video recognition with minimal computational overhead.
• To develop a data-dependent method for controlling feature interactions and routing in spatio-temporal decomposition.

Main Methods:

• GSF employs grouped spatial gating to decompose input tensors and channel weighting for fusing these decomposed tensors.
• The module adaptively routes features through time, controlling interactions in spatio-temporal decomposition.
• GSF is designed as an insertable module for existing 2D CNN architectures.

Main Results:

• GSF integration into two popular 2D CNN families resulted in efficient spatio-temporal feature extractors.
• The proposed method achieved state-of-the-art or competitive results on five standard action recognition benchmarks.
• GSF demonstrated negligible parameter and computational overhead compared to baseline models.

Conclusions:

• GSF offers an effective solution for enhancing 3D CNN performance in action recognition by addressing computational complexity.
• The module provides a flexible and efficient way to adapt 2D CNNs for video analysis tasks.
• GSF represents a significant advancement in spatio-temporal feature extraction for video understanding.