Learning View Synthesis for Desktop Telepresence With Few RGBD Cameras

Area of Science:

• Computer Vision
• Computer Graphics
• Human-Computer Interaction

Background:

• Existing telepresence systems face a trade-off between cost and quality.
• There is a need for affordable, high-quality telepresence solutions for widespread adoption.
• Commodity hardware limitations hinder the development of accessible telepresence.

Purpose of the Study:

• To develop a high-quality, cost-effective view synthesis method for telepresence.
• To enable telepresence systems to become a commodity for typical desktops.
• To present a novel neural renderer utilizing commodity RGBD cameras.

Main Methods:

• Proposed a neural renderer using a few RGBD cameras for novel view synthesis.
• Introduced Multi-Layer Point Cloud (MPC) for improved 3D reconstruction accuracy by mitigating depth camera biases.
• Implemented a temporally-aware renderer and Spatial Skip Connections (SSC) for enhanced video stability and efficient upsampling.

Main Results:

• The proposed renderer significantly outperforms recent methods in view synthesis quality.
• The method demonstrates generalization to new users and complex content like hand gestures and clothing deformation.
• Achieved high-quality results without per-video optimization, object templates, or heavy pre-processing.

Conclusions:

• The developed system offers a viable path towards commodity telepresence systems.
• The novel MPC representation and rendering techniques enable high-fidelity view synthesis from cost-effective hardware.
• The approach is robust and generalizes well, paving the way for accessible, high-quality telepresence.