Neural Projection Mapping Using Reflectance Fields.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Traditional projection mapping requires complex, time-consuming calibration.
• Achieving photorealistic rendering with dynamic lighting is challenging.

Purpose of the Study:

• To introduce a differentiable virtual projector into neural reflectance fields.
• To enable photorealistic light editing and simplify projection mapping.
• To improve scene understanding and reconstruction.

Main Methods:

• Integrating a high-resolution, spatially adaptive light source (projector) into a neural reflectance field.
• Utilizing three neural networks for geometry, material, and transmittance estimation.
• Employing an analytical Bidirectional Reflectance Distribution Function (BRDF) model and specific projection patterns for intuitive acquisition.

Main Results:

• The projected texture is fully differentiable, allowing optimization for desired appearance.
• The virtual projector simplifies calibration and improves scene understanding.
• Achieved state-of-the-art projector compensation for novel viewpoints and enhanced material/scene reconstruction.
• Demonstrated successful projection image optimization using a 2D generative model.

Conclusions:

• Neural projection mapping offers a simplified and powerful approach to augmented reality and projection mapping.
• Joint optimization of scene and projection images unlocks novel downstream applications.
• The proposed method alleviates the need for per-view calibration, making applications more accessible.