A Coding Framework and Benchmark Towards Low-Bitrate Video Understanding.

Area of Science:

• Computer Vision
• Video Processing
• Machine Learning

Background:

• Video compression is crucial for bandwidth but degrades video understanding, especially at low bitrates.
• Existing methods often fail to fully satisfy task-decoupled, label-free, and semantic prior principles for machine-friendly coding.
• Systematic investigation of compression's impact on video analysis is needed.

Purpose of the Study:

• To propose a novel traditional-neural mixed coding framework for efficient video compression.
• To address the limitations of current video compression methods in preserving semantic information for downstream tasks.
• To improve video understanding performance at low bitrates.

Main Methods:

• Developed a mixed coding framework combining traditional codecs and neural networks (NNs).
• Optimized the framework to preserve a transportation-efficient semantic representation using self-supervised learning on unlabeled data.
• Incorporated attention mechanisms and adaptive modeling to enhance semantic modeling capabilities.

Main Results:

• The proposed framework preserves rich semantics and achieves photo-realistic visual quality.
• Demonstrated empirical improvements in several mainstream downstream video analysis tasks without post-adaptation.
• The approach significantly outperforms existing methods, particularly at low bitrates.

Conclusions:

• The traditional-neural mixed coding framework effectively addresses the challenges of low-bitrate video compression for video understanding.
• The method offers a promising direction for future research in efficient and semantically-aware video compression.
• Open-sourcing of code, data, and models will facilitate further advancements in the field.