SMC++: Masked Learning of Unsupervised Video Semantic Compression.

Area of Science:

• Computer Vision
• Machine Learning
• Video Processing

Background:

• Traditional video compression prioritizes human visual perception, often leading to significant semantic loss.
• This semantic loss hinders the performance of downstream video analysis tasks.
• Existing methods struggle to balance compression efficiency with semantic preservation.

Purpose of the Study:

• To propose a video compression framework that effectively preserves semantic information.
• To develop a self-supervised approach for jointly mining and compressing video semantics.
• To mitigate the encoding of non-semantic information and semantic noise during compression.

Main Methods:

• Utilizing Masked Video Modeling (MVM) for learning generalizable semantics via masked patch prediction.
• Introducing explicit regularization on non-semantic entropy in the MVM token space.
• Instantiating the framework as Semantic-Mining-then-Compression (SMC) and an advanced SMC++ model.
• Incorporating masked motion prediction and a Transformer-based compression module in SMC++.
• Employing a compact blueprint semantic representation to align heterogeneous features.

Main Results:

• The proposed SMC and SMC++ models demonstrate superior performance compared to traditional, learnable, and perceptual quality-oriented codecs.
• Significant improvements were observed across three video analysis tasks and seven diverse datasets.
• The framework effectively preserves crucial semantic content while achieving efficient compression.

Conclusions:

• The MVM-powered compression framework offers a new paradigm for semantic-preserving video compression.
• SMC and SMC++ provide a robust solution for enhancing video analysis by maintaining semantic integrity.
• This approach advances the field by addressing the limitations of perceptual quality-focused compression methods.