UnstructuredFusion: Realtime 4D Geometry and Texture Reconstruction Using Commercial RGBD Cameras

Area of Science:

• Computer Vision
• Computer Graphics
• Human-Computer Interaction

Background:

• Professional 4D human performance capture typically requires structured multi-camera systems with extensive pre-calibration.
• These requirements limit the accessibility and practicality of such systems for everyday applications.

Purpose of the Study:

• To develop a practical, real-time, markerless human performance capture method using unstructured RGBD cameras.
• To overcome the limitations of traditional structured multi-camera setups, including hardware complexity and calibration needs.

Main Methods:

• Proposed UnstructuredFusion system utilizes three unstructured RGBD cameras with a flexible hardware setup.
• Introduced novel techniques: online multi-camera calibration, skeleton warping for non-rigid tracking, and temporal blending for atlas texturing.
• Leveraged global constraints of human body and motion via skeleton modeling and skeleton warping.

Main Results:

• Achieved high-quality 4D geometry and texture reconstruction of human activities.
• Demonstrated successful performance with flexible, even handheld, camera arrangements.
• Eliminated the need for tiresome pre-calibration and cumbersome setup associated with conventional systems.
• Addressed inherent occlusion issues present in single-camera setups.

Conclusions:

• UnstructuredFusion offers a viable and high-quality solution for 4D human performance capture using accessible hardware.
• The method significantly reduces hardware and software restrictions, promoting wider adoption.
• Enables robust reconstruction even with asynchronous camera inputs and flexible configurations.