Neural Reference Synthesis for Inter Frame Coding.

Area of Science:

• Computer Vision
• Machine Learning
• Video Compression

Background:

• Traditional video coding relies on motion estimation and compensation (MEMC), but separate optimization of enhancement and reference generation limits coding efficiency.
• Existing deep learning methods often treat reconstruction enhancement and reference generation independently, leading to suboptimal performance and potential artifacts.

Purpose of the Study:

• To introduce Neural Reference Synthesis (NRS), a novel approach for high-fidelity reference block generation in inter-frame coding.
• To collaboratively optimize reconstruction enhancement and reference generation modules for improved video compression performance.

Main Methods:

• Developed two Convolutional Neural Network (CNN) models, EnhNet for spatial reconstruction enhancement and GenNet for temporal reference synthesis.
• Implemented a new training strategy to coordinate EnhNet and GenNet, mitigating overfitting and artifacts.
• Devised a lightweight multi-level rate-distortion (R-D) selection policy for adaptive reference block selection.

Main Results:

• Achieved state-of-the-art coding gains, demonstrating over 10% Bjøntegaard Delta Rate (BD-Rate) reduction against the High Efficiency Video Coding (HEVC) anchor.
• Showcased robust performance across various video sequences and bit ranges in both low-delay and random access configurations.
• Significantly reduced computational complexity compared to existing learning-based methods through the use of lightweight deep neural networks (DNNs).

Conclusions:

• NRS offers a powerful and efficient solution for enhancing video compression through collaborative deep learning.
• The proposed training strategy and R-D selection policy ensure robust and generalizable models.
• The publicly available models facilitate reproducible research and further advancements in video coding technology.