DiffPortraitVideo: Diffusion-Based Expression-Consistent Zero-Shot Portrait Video Translation.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Zero-shot text-to-video diffusion models extend image models for video generation without retraining.
• Current methods use shape constraints and inter-frame attention, but these limit geometric deformation and texture fidelity.
• Existing approaches often exhibit flickering and inconsistent facial expressions.

Purpose of the Study:

• To introduce DiffPortraitVideo, a novel framework for high-quality, personalized text-to-video generation.
• To address limitations of existing methods, specifically improving spatio-temporal consistency and facial expression fidelity.
• To generate videos that retain textual and original image properties.

Main Methods:

• Employs a diffusion model-based feature and attention injection mechanism for key frame generation.
• Utilizes cross-frame constraints to enforce temporal coherence.
• Incorporates adaptive feature fusion to ensure consistent facial expressions.

Main Results:

• Achieves high spatio-temporal consistency and accurate expression preservation.
• Maintains fidelity to the input text and original image characteristics.
• Demonstrates superior performance compared to existing text-to-video generation methods in extensive experiments.

Conclusions:

• DiffPortraitVideo effectively generates personalized, high-quality, and coherent videos.
• The framework overcomes limitations of shape constraints and improves expression consistency.
• Presents a significant advancement in personalized text-to-video generation research.