Joint Feature Optimization and Fusion for Compressed Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Video Analysis

Background:

• Traditional action recognition methods require decoding compressed videos, which is computationally intensive.
• Existing methods struggle to effectively utilize noisy, sparse, and correlated features from motion vectors and residuals in compressed videos.

Purpose of the Study:

• To propose a joint feature optimization and fusion framework for efficient and effective action recognition directly from compressed videos.
• To address the limitations of existing methods in handling noisy and sparse motion vector and residual information.

Main Methods:

• A joint feature optimization module models feature extraction as a reconstruction process using bases for motion vectors and residuals.
• A low-rank non-local attention module is introduced to mitigate noise and sparsity during feature reconstruction.
• A lightweight feature fusion module and self-adaptive knowledge distillation are employed to generate predictions comparable to optical flow-based methods.

Main Results:

• The proposed framework achieves state-of-the-art performance on the HMDB-51 and UCF-101 action recognition benchmarks.
• The method demonstrates superior efficiency by directly processing compressed video data.
• The approach maintains a computational complexity advantage over traditional methods.

Conclusions:

• The joint feature optimization and fusion framework effectively leverages motion vectors and residuals for action recognition in compressed videos.
• This approach offers a more efficient and performant alternative to traditional video action recognition techniques.
• The proposed method sets a new standard for action recognition in compressed video analysis.