Masked Feature Residual Coding for Neural Video Compression

Area of Science:

• Computer Vision
• Machine Learning
• Video Compression

Background:

• Current neural video compression methods predict frames and use masks, but pixel-domain residuals remain significant.
• Existing methods cannot effectively leverage reconstructed temporal context for subsequent frame compression.

Purpose of the Study:

• To address limitations in pixel-domain residuals and temporal context utilization in neural video compression.
• To introduce an improved neural video compression framework called Conditional Masked Feature Residual (CMFR) Coding.

Main Methods:

• Extracted features from target and predicted frames using neural networks.
• Implemented CMFR Coding by subtracting masked predicted features from target features.
• Introduced a Scaled Feature Fusion (SFF) module for efficient conditional information removal and a Motion Refiner for enhanced optical flow quality.

Main Results:

• Achieved an 11.76% bit saving compared to a baseline model without proposed methods.
• Demonstrated significant improvements averaged over all HEVC test sequences.
• Validated the effectiveness of CMFR Coding, SFF module, and Motion Refiner.

Conclusions:

• CMFR Coding offers a more effective approach to neural video compression by operating in the feature domain.
• The proposed SFF module and Motion Refiner further enhance compression efficiency and decoded quality.
• The developed methods represent a significant advancement in neural video compression technology.