DPCS: Path Tracing-Based Differentiable Projector-Camera Systems.

Area of Science:

• Computer Vision
• Computer Graphics
• Rendering Techniques

Background:

• Projector-camera systems (ProCams) simulation is vital for spatial augmented reality (SAR) applications like relighting and compensation.
• Existing neural network methods implicitly learn scene parameters, limiting interpretability and novel scene simulation.
• Neural networks struggle with complex projection effects due to implicit indirect illumination learning.

Purpose of the Study:

• To develop a differentiable ProCams simulation method that explicitly models the physical project-and-capture process.
• To enable interpretable and efficient simulation of novel scenes using learned parameters.
• To improve the handling of complex projection effects like soft shadows and interreflections.

Main Methods:

• Introduced Differentiable Projector-Camera Systems (DPCS), a novel path tracing-based simulation approach.
• Utilized differentiable physically-based rendering (PBR) to model the project-and-capture process.
• Explicitly decoupled and learned scene parameters, integrating multi-bounce path tracing.

Main Results:

• DPCS demonstrates superior interpretability compared to previous neural network methods.
• The method efficiently handles complex interreflections and shadows.
• DPCS requires significantly fewer training samples for effective ProCams simulation.
• Achieved high-quality results in downstream ProCams tasks like relighting and compensation.

Conclusions:

• DPCS offers a physically-based, differentiable approach to ProCams simulation.
• The explicit modeling of scene parameters enhances interpretability and enables novel scene simulation.
• DPCS provides significant advantages over prior methods in terms of efficiency, accuracy, and handling of complex effects.