End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Conventional video compression (VC) methods optimize individual components, limiting end-to-end performance.
• Learned VC enables simultaneous optimization of transforms, motion, and entropy models.
• Sequential learned VC lacks the efficiency of hierarchical, bi-directional coding used in conventional VC.

Purpose of the Study:

• To develop a learned hierarchical bi-directional video codec (LHBDC).
• To combine hierarchical motion-compensated prediction with end-to-end optimization for improved video compression.
• To achieve state-of-the-art rate-distortion (R-D) performance in learned VC.

Main Methods:

• End-to-end rate-distortion optimized training of a nonlinear transform, motion, and entropy model.
• Implementation of a hierarchical bi-directional prediction structure within a learned codec framework.
• Integration of novel tools: learned masking, flow-field subsampling, and temporal flow vector prediction.

Main Results:

• Achieved state-of-the-art R-D results for learned VC in both PSNR and MS-SSIM.
• Outperformed conventional x265, SVT-HEVC, and HM 16.23 encoders in R-D performance.
• Demonstrated performance gains attributed to novel components through ablation studies.

Conclusions:

• The proposed LHBDC effectively combines hierarchical bi-directional coding with end-to-end optimization.
• LHBDC sets a new benchmark for learned video compression performance.
• The developed methods and code are publicly available for reproducibility.