Instance-Adaptive Spatial-Temporal Enhancement for Efficient Video Compression.

Area of Science:

• Computer Vision
• Video Compression
• Machine Learning

Background:

• High bitrate costs challenge efficient HD/UHD video compression.
• Existing instance-adaptive methods focus on spatial super-resolution (SR), neglecting temporal redundancy.
• Current methods suffer from excessive overfitting overheads due to limited parameter management.

Purpose of the Study:

• Introduce IASTE, the first instance-adaptive enhancement method utilizing spatial-temporal enhancement (STE).
• Incorporate low-rank adaptation (LoRA) for efficient model overfitting in video compression.
• Address the underutilization of temporal redundancy in existing video enhancement techniques.

Main Methods:

• Downscale videos spatially and temporally to reduce data volume for efficient compression.
• Overfit a specific STE model for each video using LoRA for enhancement.
• Utilize a lightweight video swin transformer model for spatiotemporal correlation capture.

Main Results:

• Achieve approximately 30% BD-Rate gains on CTC and UVG datasets compared to direct compression.
• Demonstrate about 15% gains on the YoutubeUGC dataset.
• Showcase about 10% gains on ultra-long videos in the Xiph dataset.

Conclusions:

• IASTE effectively enhances spatiotemporal resolution for efficient video compression.
• LoRA mitigates bitrate overhead for model storage by updating only low-rank matrices.
• The proposed STE method significantly outperforms existing approaches in video compression efficiency.