Cross-Modal Contrastive Masked AutoEncoder for Compressed Video Pre-Training.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Self-supervised learning (SSL) is crucial for leveraging large unlabeled video datasets.
• Compressed videos present unique challenges for representation learning due to data sparsity and noise.
• Existing methods often struggle to effectively utilize information from different modalities within compressed video.

Purpose of the Study:

• To propose a novel Transformer-based approach, Cross-modal Contrastive Masked AutoEncoder (C2MAE), for self-supervised learning on compressed videos.
• To enhance representation learning by integrating masked image modeling (MIM) and contrastive learning (CL) pretext tasks.
• To improve the handling of compressed video characteristics like I-frame sparsity and P-frame noise.

Main Methods:

• Employed a unified Transformer encoder to process visual tokens from RGB, motion vectors, and residuals.
• Developed a hybrid SSL framework combining MIM and CL, extending VideoMAE with Fine-Grained Motion-aware Masking (FGMM) and Multi-modal Reconstruction (MR).
• Introduced Contrastive Cross-modal Learning (CCL) by comparing features from compressed and raw video clips.

Main Results:

• C2MAE significantly enhances cross-modal interactions, effectively compensating for the limitations of compressed video data.
• Achieved state-of-the-art results on UCF-101, HMDB-51, and Kinetics-400 benchmarks.
• Demonstrated the effectiveness of C2MAE in delivering stronger pre-trained models for video understanding tasks.

Conclusions:

• C2MAE offers a powerful and effective framework for self-supervised learning on compressed videos.
• The proposed FGMM strategy and CCL module contribute to superior representation learning.
• The approach successfully addresses the challenges posed by compressed video data, leading to significant performance gains.