HVTR++: Image and Pose Driven Human Avatars Using Hybrid Volumetric-Textural Rendering.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Neural rendering methods excel at photorealistic human avatar generation.
• Existing methods struggle with detailed appearance due to limited driving signals like body poses.
• Low-dimensional signals fail to capture the full complexity of clothed human appearance.

Purpose of the Study:

• To develop a novel neural rendering pipeline for synthesizing high-quality 3D human avatars.
• To improve the faithfulness of avatar details by incorporating driving view images.
• To achieve efficient and high-quality avatar synthesis from arbitrary poses and views.

Main Methods:

• Proposed Hybrid Volumetric-Textural Rendering (HVTR++) pipeline.
• Learned pose and view image signals on a dense UV manifold for UV-aligned features.
• Constructed a 3D volumetric representation using a pose- and image-conditioned neural radiance field (PID-NeRF).
• Fused volumetric features with 2D textural features in image space.
• Employed a fast Generative Adversarial Network (GAN)-based textural renderer for high-resolution output.

Main Results:

• HVTR++ successfully synthesizes detailed 3D human avatars from arbitrary poses and views.
• The hybrid rendering approach handles complex motions and self-occlusions effectively.
• Achieved efficient inference times with high-quality avatar generation.
• Demonstrated state-of-the-art quantitative results compared to existing methods.

Conclusions:

• HVTR++ offers an efficient and high-quality solution for 3D human avatar synthesis.
• The integration of volumetric and textural rendering overcomes limitations of previous methods.
• The pipeline effectively preserves appearance details and handles complex dynamics.