UniFES: A Unified Recurrent Network for Quality Enhancement and Stabilization in Face Videos

Area of Science:

• Computer Vision
• Video Processing
• Artificial Intelligence

Background:

• Face content is increasingly shifting from static images to dynamic videos.
• This shift introduces complex artifacts, combining pixel-wise distortions with motion-related impairments.
• Existing methods struggle to address these intertwined spatial-temporal challenges in face videos.

Purpose of the Study:

• To propose a unified framework for joint face video quality enhancement and stabilization.
• To develop a method that effectively aggregates information from both pixel and motion domains.
• To address the limitations of current approaches in handling the dual challenges of video quality and motion.

Main Methods:

• Introduced UniFES (Unified recurrent network for joint Face video quality Enhancement and Stabilization), a novel recurrent network.
• Decomposed temporal alignment for quality enhancement using progressive feature alignment with explicit physical information (global dynamics).
• Integrated mixed dynamics from enhancement (pixel domain) into stabilization for robust trajectory estimation and refined warping masks for rendering.

Main Results:

• UniFES demonstrates superior performance in joint face video quality enhancement and stabilization.
• Achieved state-of-the-art results compared to 32 baseline methods on both synthetic and real-world datasets.
• Successfully established a new synthetic dataset for training and evaluating this emerging task.

Conclusions:

• UniFES is the first successful attempt at joint face video quality enhancement and motion stabilization.
• The proposed method effectively leverages mutual information across pixel and motion domains.
• UniFES offers a robust solution for improving the quality and stability of dynamic face videos.