Multi-Modality Deep Restoration of Extremely Compressed Face Videos

Area of Science:

• Computer Vision
• Signal Processing
• Artificial Intelligence

Background:

• Talking head videos are ubiquitous in digital communication.
• Compression artifacts degrade video quality, especially for faces.
• Existing methods struggle with aggressive compression.

Purpose of the Study:

• To develop a deep learning method for restoring aggressively compressed talking head videos.
• To improve the visual quality of face videos in low-bandwidth scenarios.

Main Methods:

• A multi-modality deep convolutional neural network (DCNN) was developed.
• The DCNN architecture incorporates priors from synchronized speech signals.
• Semantic elements from the compression code stream (motion vectors, partition maps, quantization parameters) were integrated.

Main Results:

• The proposed DCNN method demonstrates superior performance in restoring face videos.
• Empirical evidence validates the effectiveness over state-of-the-art methods.
• The integration of multi-modality priors significantly enhances artifact removal.

Conclusions:

• The novel DCNN approach effectively restores aggressively compressed talking head videos.
• Incorporating speech and compression code stream priors improves deep learning-based video restoration.
• This method offers a significant advancement for video communication quality.