Robust Pose Transfer With Dynamic Details Using Neural Video Rendering.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Human pose transfer aims to generate realistic videos of a target person mimicking a source person's actions.
• Existing methods often require extensive training data, limiting their effectiveness on accessible internet videos.
• Performance degradation occurs with insufficient training frames in current pose transfer techniques.

Purpose of the Study:

• To develop a neural video rendering framework capable of preserving dynamic details from short monocular videos.
• To improve the robustness and fidelity of human pose transfer, especially with limited training data.
• To address the limitations of existing methods that rely on large datasets.

Main Methods:

• Proposed a neural video rendering framework integrated with a dynamic details generation network (D²G-Net).
• Introduced a novel hybrid texture representation to encode static and pose-varying appearance characteristics.
• Mapped hybrid texture to image space for detail-rich frame rendering in the neural rendering stage.

Main Results:

• Demonstrated preservation of dynamic details even when trained on short videos (2k-4k frames).
• Achieved clearer dynamic details compared to existing methods.
• Showcased more robust performance on accessible short videos.

Conclusions:

• The proposed framework effectively utilizes both 3D feature stability and learning-based components.
• The hybrid texture representation enables detailed and accurate rendering of pose-varying appearances.
• The method offers a significant advancement in human video pose transfer using limited data.