DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems.

Area of Science:

• Computer Vision
• Computer Graphics
• Robotics

Background:

• Projector-camera systems (ProCams) are crucial for spatial augmented reality (SAR).
• Traditional ProCam tasks like relighting, compensation, and reconstruction are handled independently, creating practical challenges for SAR.
• Existing methods often require specific devices, calibrations, and multiple image samples.

Purpose of the Study:

• To propose a novel, end-to-end trainable model, DeProCams, for unified ProCam tasks.
• To simultaneously address image-based relighting, projector compensation, and depth/normal reconstruction.
• To simplify SAR applications by integrating multiple ProCam tasks into a single model.

Main Methods:

• DeProCams decomposes ProCam mappings into shading estimation, direct light estimation, and neural rendering.
• It incorporates a differentiable projector direct light mask to handle occlusions using epipolar constraints.
• Photometric and geometric constraints are applied for improved convergence and plausible intermediate results.

Main Results:

• DeProCams achieves state-of-the-art performance in quality for the three core ProCam tasks.
• The model is fully differentiable, enabling end-to-end training.
• Experimental results demonstrate clear advantages over previous methods.

Conclusions:

• DeProCams offers a unified solution for multiple ProCam tasks, reducing complexity for SAR.
• It eliminates the need for additional optical devices, radiometric calibrations, and structured light.
• The model provides a more efficient and integrated approach to ProCam applications.