Video Action Recognition Using Motion and Multi-View Excitation with Temporal Aggregation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Video action recognition relies on effective spatiotemporal and motion feature representations.
• Existing methods like 3D CNNs are computationally expensive, while (1+2)D CNNs may neglect motion information.

Purpose of the Study:

• To propose a novel building block, the META block, for efficient and faithful capture of spatial, temporal, and motion features in videos.
• To address the limitations of existing 3D CNN and (1+2)D CNN approaches in video action recognition.

Main Methods:

• Introduction of the META block, comprising Motion Excitation (ME), Multi-view Excitation (MvE), and Densely Connected Temporal Aggregation (DCTA).
• ME encodes feature-level frame differences.
• MvE enriches spatiotemporal features using adaptive multi-view representations.
• DCTA models long-range temporal dependencies.
• Integration of the META block into a 2D ResNet-50 architecture.

Main Results:

• The proposed META block architecture demonstrates superior performance over previous CNN-based methods on the Something-Something v1 and Jester datasets, measured by 'Val Top-1 %'.
• Competitive results were achieved on the Moment-in-Time Mini dataset.

Conclusions:

• The META block offers a more faithful and efficient method for capturing spatial, temporal, and motion features in video action recognition.
• The proposed approach represents a significant advancement over existing CNN-based techniques for video understanding tasks.