Self-Supervised Learning of Perceptually Optimized Block Motion Estimates for Video Compression.

Area of Science:

• Computer Vision
• Video Compression
• Machine Learning

Background:

• Block-based motion estimation is crucial for video codecs but computationally intensive and prone to aperture problems.
• Current methods lack optimization for perceptual quality of decoded images.
• Existing block matching criteria do not consider the visual fidelity of motion-compensated frames.

Purpose of the Study:

• To develop a search-free block motion estimation framework for perceptually optimized video compression.
• To introduce a multi-stage convolutional neural network for simultaneous multi-block size motion estimation.
• To improve the computational efficiency and perceptual quality of inter prediction in video codecs.

Main Methods:

• A composite block translation network (CBT-Net) utilizing a triplet of frames as input.
• Self-supervised training on a large dataset of uncompressed video content.
• Employing the multi-scale structural similarity (MS-SSIM) loss function for perceptual quality optimization.

Main Results:

• The CBT-Net demonstrates computational efficiency compared to traditional block matching algorithms with similar prediction errors.
• Integration into AV1 inter prediction yields average Bjøntegaard-delta rate (BD-rate) improvements of -1.73% (MS-SSIM) and -1.31% (VMAF).
• The proposed method achieves comparable prediction errors with significantly reduced computational load.

Conclusions:

• The search-free CBT-Net offers an efficient and perceptually optimized alternative to conventional block motion estimation.
• This deep learning approach advances the goal of perceptually optimized motion estimation in video coding.
• The framework shows significant improvements in video quality metrics when integrated into modern video encoders like AV1.