NerfCap: Human Performance Capture With Dynamic Neural Radiance Fields.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Human performance capture from videos is challenging due to limitations of existing methods like silhouette-based or photometric alignment.
• These methods struggle with detailed surface deformation and are unstable under appearance variations.

Purpose of the Study:

• To propose NerfCap, a novel performance capture method using dynamic neural radiance fields (NeRF).
• To overcome the limitations of existing mesh-based approaches for capturing complex human motion and appearance.

Main Methods:

• NerfCap initializes a canonical NeRF from a template mesh and optimizes its deformation and appearance models to align with video frames.
• It combines linear blend skinning with embedded graph deformation to capture both large body motions and fine surface details.
• The method can be pre-trained end-to-end in a self-supervised manner by matching synthesized and input videos.

Main Results:

• NerfCap achieves superior surface reconstruction accuracy compared to prior works.
• It demonstrates enhanced novel-view synthesis quality, producing more photorealistic images.
• The method flexibly captures complex geometry and appearance variations present in videos.

Conclusions:

• NerfCap offers a robust and flexible solution for human performance capture from sparse video data.
• The dynamic NeRF representation enables more accurate and detailed reconstruction than traditional mesh-based methods.
• NerfCap advances the state-of-the-art in generating realistic human performances from video.